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FOR EWORD 


The  growth  of  modern  Naval  power  abroad  along  with  other 
threats  to  our  national  security  has  brought  about  a  spiraling  increase 
in  complexity  of  Naval  systems  to  meet  increasing  levels  of  threat. 

This  has  resulted  in  problems  of  achievable  operational  effectiveness  * 

in  an  era  when  effectiveness  is  more  crucial  than  ever  before.  At  the 
same  time,  economic  pressures  at  home  require  that  a  maximum  pay¬ 
off  in  effectiveness  per  dollar  be  achieved  and  that  the  investment  of 
other  resources  be  realistic.  To  achieve  this,  our  full  potential  for 
designing  effectiveness  into  systems  must  be  realized. 

The  Navy's  investment  into  the  development  of  analytic  and 
management  design-decision  techniques  for  ensuring  required  levels 
of  effectiveness  has  been  substantial  over  the  past  decade.  Consider¬ 
able  progress  has  been  made  under  the  Systems  Performance  Effec¬ 
tiveness  (SPE)  program  in  developing  useful  approaches,  techniques 
and  tools  for  system  effectiveness  planning,  design  and  evaluation. 

This  manual  summarizes  the  Navy's  current  effectiveness  concepts 
and  the  tools  presently  available,  and  discusses  their  application  to 
the  Navy's  acquisition  process.  Navy  program  managers,  project 
managers  and  project  engineers  are  urged  to  use  this  manual  as  a  guide 
to  designing  for  increased  effectiveness  through  application  of  the 
"effectiveness  approach"  and  the  tools  available  for  its  implementation 
in  system  engineering  programs. 

The  first  edition  of  this  manual  was  published  in  1967  (NAVMAT 
P3941),  followed  by  an  updated  version  in  July  1968  (NAVMAT  P3941-A). 

Since  then,  the  continued  development  of  tools  and  procedures  together 
with  further  experience  in  their  application  has  created  the  need  for 
substantial  revision.  As  a  consequence,  the  manual  has  been  rewritten 
to  incorporate  new  material  to  meet  current  needs  within  the  Naval 
Material  Command.  Comments  or  recommendations  concerning  the 
contents  of  this  manual  should  be  forwarded  to  Code  4100,  Naval  Elec¬ 
tronics  Laboratory  Center,  San  Diego,  California  92152. 


ADMINISTRATIVE  INFORMATION 


This  manual  was  prepared  for  the  Naval  Electronics  laboratory 
Center  by  D.  T.  Ranifan  of  The  Harbinger  Corporation,  Santa  Monica, 
California  under  Contract  N0024-73-R-0087,  under  the  technical  direc¬ 
tion  of  H.  Greenstein,  Product  Assurance  Division  (Code  4100)  in 
partial  fulfillment  of  the  Division's  responsibilities  under  the  Systems 
Performance  Effectiveness  Program.  Work  was  performed  under 
Project  SF099- 90-002,  Task  Area  1622  (NELC  R124),  during  the 
period  July  1971  -  September  1973. 
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CHAPTER  ONE:  INTRODUCTION 


1 .  1  Purpose  of  the  Manual 

This  manual  is  not  intended  to  be  a  training  manual  or  technical 
reference  work  for  system  effectiveness  techniques.  Rather,  it  is 
offered  as  a  general  guide  to  concepts  and  tools  for  Navy  and  contractor 
program  managers,  project  managers  and  project  engineers.  Its 
specific  purposes  are  to 

•  provide  an  overview  of  the  Navy's  current  concepts  of 
system  effectiveness  and  its  elements, 

•  provide  management-level  guidance  in  implementing 
system  effectiveness, 

•  show  how  system  effectiveness  aids  the  decision-making 
process,  and 

•  summarize  tools  and  techniques  currently  available  for 
use  in  system  acquisition  programs. 

Although  the  literature  on  system  effectiveness  contains  varying 
formulations  and  definitions  of  terms,  no  systematic  attempt  is  made 
here  to  review  and  compare  the  various  approaches  and  definitions 
used  by  different  authors  and  organizations.  The.  approach  taken  is 
to  select  and  present,  to  the  extent  possible,  a  consistent  body  of 
approaches,  definitions,  formulations  and  categorizations  for  use  in 
Navy  system  effectiveness  programs. 

It  is  hoped  that  greater  familiarity  with  the  concepts  and  tools 
of  system  effectiveness  analysis  will  lead  to  more-widespread  imple¬ 
mentation  in  Navy  programs- -and  to  realization  of  the  increased 
effectiveness  per  dollar  for  Navy  systems  that  can  be  achieved. 


This  manual  is  supported  by  detailed  procedure-  and  methodology- 
manuals  at  both  NAYMAT  and  using  command  levels  for  application  of 
the  various  subdisciplines  of  effectiveness  (reliability,  maintainability, 
etc.  ). 


1 . 2  Application 

The  approach  outlined  in  this  manual  is  applicable  to  all  Navy 
systems  throughout  the  acquisition  life  cycle  from  Concept  Formulation 
through  Operational  Phases,  although  major  emphasis  is  placed  here  on 
Concept  Formulation,  Validation  and  Full  Scale  Development.  The 
criticality  of  conducting  system  effectiveness  analysis  depends  on  the 
level  of  study  and  design  effort  and  becomes  more  significant  for  more- 
complex  system  and  total  force  studies.  The  approach  should  be  spec¬ 
ified  whenever  operational  effectiveness  or  system  value  is  considered 
to  be  an  important  basis  for  selection  of  system  design  and/or  support 
system  alternatives,  or  when  system  optimization  comprises  a  signi¬ 
ficant  effort. 

System  effectiveness  analysis  should  be  applied  by  NAVMAT  and 
using  commands  for  generating  requirements  and  selecting  major  sys¬ 
tem  options.  Prime  and  associate  contractors  and  major  subcontractors 
should  apply  the  techniques  for  optimization  of  major  systems/ 
subsystems. 

The  techniques  discussed  in  this  manual  are  flexible  and  may  be 
tailored  to  a  range  of  budget  and  schedule  requirements,  although  the 
useful  level  of  analysis  will  depend  on  the  sensitivity  of  life  cycle  cost 
and  system  value  to  variation  in  major  effectiveness  variables,  on  the 
criticality  of  the  system  in  the  Navy's  overall  weapon  system  program, 
and  on  the  degree  of  risk  inherent  in  the  acquisition  program  for  the 
system  in  question.  The  system  effectiveness /system  value  (SE/SV) 
methodology  described  herein  is  specifically  aimed  at  reducing  system 
development  program  risks  through  rational  decision-making  processes 
which  match  total  system  characteristics  to  operational  requirements 
and  conditions  well  in  advance  of  deployment. 

1 . 3  Benefits  to  Management 

The  modern  concepts  of  system  effectiveness  analysis  grew  out 
of  the  need  for  a  rational  basis  for  making  design  decisions  in  complex 
system  acquisition  programs.  Since,  from  economic  and  scheduling 
necessities,  design  hypotheses  must  be  tested  before  major  invest¬ 
ments  are  made,  an  analytic  structure  is  required  within  which  all 
elements  of  the  system  can  be  related  to  each  other  and  to  the  whole 
in  terms  of  their  contribution  to  potential  effectiveness.  Furthermore, 


since  resources  are  limited  and  lower-level  design  goals  are  often 
conflicting,  compromises  or  trade-offs  are  required,  and  this  requires 
a  common  measure  or  criterion.  In  addition,  a  framework  is  required 
which  facilitates  communication  among  the  technical  and  management 
members  of  the  design  team.  Obviously,  if  a  model  of  the  system  can 
be  manipulated  to  measure  the  system  rather  than  the  system  itself, 
considerable  savings  in  time  and  resources  can  be  achieved. 

Effectiveness  analysis  provides  a  framework  and  method  for 
selecting  and  optimizing  systems  within  the  broader  context  of  missions 
and  objectives,  and  has  evolved  into  a  central  tool  for  engineering  and 
management  decision-making.  Within  the  inherent  limitations  of  any 
analytical  approach,  used  appropriately  during  the  system  engineering 
process  the  system  effectiveness  approach  provides  one  of  the  critical 
ingredients  for: 

producing  a  system  within,  acceptable,  defined  limitations  of 

•  time 

•  resources  of  dollars,  management,  manpower  and 
facilities 

•  technology  and  ^tate-of-the-art 

which  performs  its  assigned  mission(s)  with  acceptable  effec¬ 
tiveness  and  cost 

•  under  stated  environmental  conditions 

•  within  an  established  mission-time  frame 
when  deployed  and  operated  under  conditions  of 

•  defined  types  and  levels  of  support 

•  established  policies  and  doctrine 

•  interaction  with  other  systems  with  which  it  must 
interface 
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The  benefits  to  management  of  implementing  a  systematic  pro¬ 
gram  of  system  effectiveness  analysis  increase  with  the  complexity 
of  the  mission,  system  and  acquisition  program.  These  benefits 
include  the  following: 

•  Management  can  exercise  oetter  control  since  prediction 
of  the  system's  effectiveness  is  in  a  form  which  enables 
potential  trouble- spots  to  be  pinpointed  early,  and  pro¬ 
vides  visibility  to  aspects  of  the  design  process  which 
are  not  otherwise  visible  until  much  later. 


Accountability  is  improved  since  an  objective,  quanti¬ 
tative  measure  of  effectiveness  for  major  elements 
provides  an  important  measure  of  performance  of 
the  group  responsible. 

System  effectiveness  formulation  provides  a  rational 
and  readily- communicated  vehicle  for  stating  require¬ 
ments,  and  thus  reduces  the  communication  gap  between 
user  and  producer  within  the  Navy,  Navy  and  contractor, 
contractor  and  subcontractor,  and  among  personnel 
within  all  organizations. 

The  iterations  required  to  produce  an  effective  final 
design  may  be  reduced  since  the  interactions  of  major 
design  factors  and  system  elements  are  explicit  and 
more-readily  comprehended. 

A  basis  is  provided  for  determining  the  sensitivity  of 
overall  effectiveness  to  the  contributions  of  the  various 
design  factors,  thus  providing  an  objective  means  for 
allocating  resources  c.mong  competing  design  and 
"ility"  groups  or  functions. 

When  associated  with  cost,  schedule,  and  mission 
analysis  (which  establishes  worth  and  utilization  factors 
as  well  as  the  effectiveness  criteria),  effectiveness 
analysis  and  evaluation  of  competing  concepts,  designs 
or  plans  provide  the  most  complete  and  credible  basis 
available  today  for  deciding  among  alternatives  and 
optimizing  total  system  design  in  terms  of  overall 
objectives  and  resources. 


Ultimately,  of  course,  all  decisions  depend  on  the  judgment  of 
responsible  decision  makers  and  depend  heavily  on  many  factors  which 
are  only  marginally  or  not  at  all  susceptible  to  quantitative  analysis. 
Furthermore,  even  the  best  quantitative  analyses  in  the  "real  world" 
are  subject  to  uncertainties  and  risks  (which  are  also  estimated  in  an 
adequate  analysis).  Hence,  effectiveness  analysis  does  not  produce 
decisions  but  is  simply  one  of  the  bases  for  decision.  Nevertheless, 
it  is  an  extremely  valuable  tool  for  decision  since  it  involves  a  sys¬ 
tematic  examination  of  the  possibilities,  and  provides  a  rational  way 
of  estimating  the  consequences  of  the  interaction  of  systems  of  vari¬ 
ables  which  would  be  difficult  or  impossible  to  accomplish  on  an 
intuitive  basis. 


In  summary  ".  .  .  the  virtue  of  analysis  is  that  it  is  able  to  make 
a  more  systematic  and  efficient  use  of  judgment  than  any  of  its  alter¬ 
natives.  The  essence  of  the  method  is  to  construct  and  operate  within 
a  "model"- -an  idealization  of  the  situation  appropriate  to  the  problem. 
Such  a  model  .  .  .  introduces  a  precise  structure  and  terminology  that 
serve  primarily  as  a  means  of  communication,  enabling  the  partici¬ 
pants  in  the  study  to  make  their  judgments  in  a  concrete  context.  More¬ 
over,  through  feedback.  .  .  the  model  helps  the  decision-maker,  the 
analysts,  and  the  experts  on  whom  they  depend  to  revise  their  earlier 
judgments  and  thus  to  arrive  at  a  clearer  understanding  of  the  problem 
and  its  context.  "* 


Quade,  Edward  S.  ,  "Introduction  and  Overview"  in  Goldman,  Thomas  A. 
(Ed.),  Cost  Effectiveness  Analysis,  Praeger,  N.Y.,  1967,  pg.  3-4. 
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CHAPTER  TWO:  THE  CONCEPT  OF  SYSTEM  EFFECTIVENESS 


2.  1  System  Effectiveness:  An  Element  cf  System  Value 

Selection  and  optimization  of  a  system  require  that  competing 
concepts  and  designs  be  considered  in  relation  to  each  other  and  to 
their  uses  in  order  to  acquire  the  system  which  has  the  greatest 
operational  value.  Thus,  an  important  decision  criterion  is  an  index 
of  value  associated  with  competing  configurations  that  accounts  for 
and  balances  the  system  credit  and  debit  factors  as  well  as  the  relative 
frequencies  and  worths  of  the  various  uses  of  the  system.  On  the 
credit  side  is  the  effectiveness  of  the  system  with  respect  to  each  of 
its  uses  (missions).  On  the  debit  side  is  the  cost  or  penalty  associated 
with  achieving  that  effectiveness,  including  acquisition  and  operational 
dollar  costs,  acquisition  time  and  resource  costs,  and  penalty  costs 
to  other  systems  through  such  things  as  competitive  use  of  support 
systems.  Where  multiple  missions  exist  for  a  system,  the  relative 
frequency  and  military  worth  of  each  mi33ion  must  also  be  considered. 
If  the  military  worth  of  missions,  effectiveness  of  systems,  or  penal¬ 
ties  are  expected  to  change  over  the  system's  life  cycle,  then  "degra¬ 
dation"  or  "improvement"*  factors  also  need  to  be  accounted  for  -- 
particularly  if  rate  of  degradation  or  improvement  varies  among  the 
various  missions  or  competing  system  concepts.  A  classification  of 
these  factors  is  shown  diagrammatically  in  Figure  2-1.  An  early 
formulation  for  relating  some  of  the  factors  quantitatively  is  the  value 
index  given  by  Equation  2-1  of  Figure  2-2  --  the  so-called  Index  of 
Navy  Defense  Effectiveness,  Ej. 

Actually,  Equation  2-1  is  illustrative  and  would  be  applicable  as 
given  only  in  special  cases.  In  application,  the  formulation  E(j  would 
be  made  in  light  of  the  particulars  of  the  systems  and  missions  being 
considered.  In  illustration,  a  somewhat  more- general  formulation  is 


The  concept  of  system  degradation  as  a  function  of  time  is  familiar 
to  everyone.  The  possibility  of  "improvement"  also  exists,  and 
examples  are  common  of  such  system  "learning"  processes  as  "avail¬ 
ability  growth"  resulting  from  elimination  of  early  failures  of  defec¬ 
tive  parts,  correction  of  errors  in  maintenance  and  operating  data, 
shaking  down  of  the  logistic  support  system,  technician  learning  through  on- 
the-job  experience,  design  improvements,  and  the  like. 
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Figure  2-1.  Factors  to  be  considered  in  formulating  an  index  of  system 
operational  value 


provided  by  Equation  2-2  of  Figure  2-2.  However,  in  many  cases 
cost  and  effectiveness  terms  are  interdependent  and  a  simplistic  use 
of  such  equations  is  riot  possible.  Actually,  to  obtain  a  valid  decision 
index,  before  applying  a  formulation  of  this  type  each  system  being  eval¬ 
uated  should  have  been  optimized  such  that  the  terms  represent  optimal 
values  in  a  "cost-effectiveness"  sense.  For  instance,  effectiveness 
would  have  been  defined  as  functions  of  acquisition  dollar,  time  and 
resource  costs  as  well  as  operational  dollar  and  other  penalty  costs. 
These  functions  would  then  have  been  used  to  determine  the  rnost- 
favorable  (optimal)  values  of  effectiveness  and  costs.  These  values 
would  then  be  used  along  with  the  other  required  information  to  deter¬ 
mine  the  overall  index  of  value  for  each  system. 

From  the  foregoing  discussion,  it  can  be  seen  that  a  quantitative 
measure  of  system  effectiveness  is  essential  if  the  relative  "values" 
of  competing  system  concepts  and  designs  are  to  be  estimated.  In  the 
next  section,  the  system  effectiveness  term  is  examined  in  more  detail. 

2.2  Effectiveness:  Net  Usable  Performance  Related  to  Required  Level 

System  Effectiveness  (Es)  can  be  defined  as  "a  measure  of  the 
ext'- at  to  which  a  system  can  be  expected  to  complete  its  assigned 
mission  within  an  established  time  frame  under  stated  environmental 
conditions.  "*  Although  various  figures  of  merit  have  been  used  to 
represent  this  measure,  the  most  generally  useful  and  directly  inter¬ 
pretable  measures  are  quantitative  and  stated  in  probabilistic  terms. 
Thus,  system  effective:. ess  may  be  defined  more  specifically  as  "the 
probability  that  a  system  can  successfully  meet  an  operational  demand 
throughout  a  given  time  period  when  operated  under  specified 
conditions .  "* 

In  addition  to  probability  of  success,  the  performance -requirement 
ratio  is  also  useful  since  this  may  give  some  insight  into  the  degree  by 
which  the  requirement  is  not  met  or  is  exceeded  --  which  may  be  an 
important  consideration,  particularly  if  appreciable  risk  and  uncertainty 
exist  in  the  estimate  of  the  system's  performance  or  if  the  validity  of 
the  requirement  is  in  question.  However,  such  ratios  need  to  be  inter¬ 
preted  with  caution  as  will  be  shown. 


ft 

Systems  Effectiveness,  compiled  by  Systems  Effectiveness  Branch, 
Office  of  Naval  Material,  January  1965  (AD  659-520). 
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Before  proceeding  further,  it  will  be  helpful  to  discuss  the 
meaning  of  performance,  capability  and  related  terms  as  used  in  this 
manual.  The  term  "performance, "  when  used  to  describe  the  output 
of  a  system,  is  meant  in  its  broadest  sense  and  includes  the  total  set 
of  usable  outputs.  Thus,  the  performance  of  a  system  at  any  given 
time  may  vary  anywhere  from  the  maximum  potential  performance  of 
which  it  is  capable  under  the  most  favorable  conditions,  through  various 
reduced  levels  resulting  from  extrinsic  or  intrinsic  factors,  to  zero 
performance  (as  in  system  failure).  Unfortunately,  it  has  also  become 
common  practice  to  use  the  term  "performance"  as  a  short-hand  term 
in  lieu  of  "maximum  (potential)  performance,  "  sometimes  called  "per¬ 
formance  capability,  "  which  assumes  dependable  operation  in  a  spec¬ 
ified  environment  as  will  be  discussed  later.  In  some  instances,  this 
manual  also  uses  the  term  performance  in  this  way,  but  the  meaning 
should  be  clear  from  context.  In  any  case,  however,  the  term  per-r 
iormance  refers  to  the  actual  level  or  amount  of  output(s)  of  a  system 
without  regard  to  the  level  or  amount  required.  Further,  unless  clear 
from  context,  the  term  "performance"  is  used  in  the  sense  of  net  or 
usable  output  where  reliability,  support  system,  environment  and 
other  factors  are  taken  into  account. 

In  no  case  are  the  terms  "performance"  and  "Capability"  to  be  used 
synonymously.  As  discussed  later,  the  term  Capability  is  defined 
within  3  system  effectiveness  modeling  context  as  the  "probability  that 
the  system1  s  designed  (or  maximum)  performance  will  allow  it  to  meet 
mission  demands  successfully  assuming  that  the  system  is  available 
and  dependable.  "  Thus,  Capability  implies  a  comparison  of  (1)  designed 
performance  (the  required  minimum  performance)  with  (Z)  the  level  of 
performance  needed  to  carry  out  a  mission  under  consideration  -- 
whether  or  not  these  demands  are  within  the  original  system  specifi¬ 
cations  . 

Preserving  the  distinction  between  Capability  and  performance, 
and  distinguishing  between  the  general  use  of  the  term  performance  to 
denote  actual  nir  usable  output  at  a  given  time  and  its  special  use  to 
denote  maximum  or  designed  performance,  will  reduce  much  potential 
,  onfusion. 


2.2.1  Effectiveness  vs.  Usable  Performance 

In  some  cases  the  requirement  (effectiveness  criterion)  may  be 
difficult  to  establish  because  an  analysis  of  higher-level  objectives 
involving  missions  and  systems  outside  the  scope  of  a  particular  design 
effort  has  not  been  accomplished.  In  such  cases,  the  success 
criterion  either  has  a  certain  arbitrariness  or  simply  is  not  available,  and 
a  system  performance  measure  is  all  that  can  be  reported.  In  that 
case,  either  the  system  with  the  greatest  performance  would  be  selec¬ 
ted  or,  perhaps,  the  one  which  provides  the  maximum  performance 
per  cost.  However,  such  measures  should  not  be  reported  as  "effec¬ 
tiveness"  values  since  they  are  not  related  to  a  valid  required  level  of 
performance  derived  from  objectives.  The  existence  of  such  situations 
does  not  invalidate  the  effectiveness  concept,  but  rather  points  to  the 
need  for  appropriate  higher-level  studies  by  the  user  for  establishing 
meaningful  requirements . 

Examples  of  system  performance  measures  which  have  frequently 
been  reported  without  reference  to  a  required  level  include  percent  tar¬ 
get  destruction,  circular  error  probability,  channel  capacity,  probability 
of  error,  ton-miles /day ,  passenger-miles /day,  etc.  In  all  such  cases, 
higher-level  analyses  need  to  be  performed  to  establish  the  criteria  for 
acceptable  level  of  system  performance  before  statements  can  be  made 
about  the  system's  effectiveness  as  opposed  to  performance. 

To  illustrate  the  difference  between  measures  of  performance 
and  effectiveness,  suppose  a  transportation  system  is  being  evaluated, 
and  that  its  net  output  is  measured  in  ton-miles /day.  What  is  its 
effectiveness  --  i.  e.  ,  how  effective  is  it? 

While  it  makes  sense  to  say  that  a  system  which  is  capable  of 
transporting,  say,  an  average  of  100  ton-miles/day  is  more  effective 
than  one  which  transports  only  50,  we  don't  know  how  effective  it  is 
unless  a  goal  has  been  specified  --  nor  do  we  know,  for  irstance, 
whether  it  has  been  over-  or  under-designed.  It  makes  no  sense  to 
say  that  the  system's  effectiveness  is  100  ton -miles /day.  Strictly 
speaking,  the  100  ton-miles /day  is  its  average  performance. 


To  determine  the  system's  effectiveness,  we  would  need  to  know 
how  many  ton-miles/day  are  required  and,  since  the  svstem's  per¬ 
formance  may  vary  depending  on  numerous  factors,  we  need  to  know 
more  about  it  than  its  average,  such  as  the  distribution  of  ton-miles/ 
day.  Thus,  we  might  determine  that  the  requirement  for  a  given 


mission  is  150  ton-miles /day  and  that,  while  the  average  performance 
is  100  ton -miles /day,  its  probability  distribution  of  ton-miles/day  is 
given  by  curve  A  in  Figure  2-3.  v  According  to  the  curve,  the  system 
will  transport  150  ton-miles/day  or  greater  with  a  probability  of  0.25. 
Thus,  its  probability  of  ’’success,  "  which  tellt  us  how  effective  the 
system  is  likely  to  be  in  meeting  its  requirement,  is  0.25  with  respect 
to  the  given  mission.  Performance  is  specified  by  the  "probability 
distribution"  of  ton-miles/day,  the  effectiveness  criterion  or  objective 
is  "deliver  at  least  150  ton  -.miles /day,  "  and  its  effectiveness  is  0.  25, 
which  is  the  probability  of  meeting  the  criterion  (meeting  the  mission 
demands).  Another  measure  which  might  be  of  interest  is  its  average 
performance -requirement  ratio,  which  is  100/150  =  0.67.  This  has 
been  interpreted  as  showing  the  extent  to  which  a  system  meets  its 
requirement,  but  as  will  be  seen  this  is  not  necessarily  a  good  measure 
of  "extent"  since  it  can  be  misleading. 

An  effectiveness  of  0.  25  is  quite  low  and  we  would  probably  con¬ 
clude  that  the  system  is  unacceptable.  If  another  system  with  greater 
effectiveness  were  not  available,  either  an  effectiveness  improvement 
program  or  design  of  a  new  system  would  be  indicated,  depending  on 
cost  and  lead-time  factors.  On  the  other  hand,  if  the  type  of  mission 
permitted,  two  "systems"  might  be  used  since  the  average  performance 
requirement  ratio  would  then  be  200/150  =  1.33,  which  would  appear  to 
provide  an  acceptable  margin  over  the  requirement.  In  fact,  however, 
under  assumptions  of  independence,  the  distribution  of  performance 
for  two  such  "systems"  operating  together  (when  combined  under  the 
appropriate  rules  of  mathematical  statistics)  is  as  shown  in  curve  B 
of  Figure  2-3.  According  to  curve  B,  two  systems  will  transport 
150  ton-miles/day  or  more  with  a  probability  of  0.  68,  the  new  level  of 
effectiveness.  Curve  C  is  the  three -system  case,  which  has  an  effec¬ 
tiveness  of  0.88,  which  is  still  not  a  particularly  high  probability  of 
success  --  even  though  the  average  performance- requirement  ratio  is 
300/150  =  2.  0. 

From  the  foregoing,  it  is  seen  that  using  average  performance 
in  the  performance-requirement  ratio  led  to  easily  misinterpreted 
results.  It  was  used  in  the  illustration  because  of  its  use  by  some 


Strictly  speaking,  a  probability  distribution  as  ordinarily  defined 
expresses  the  probability  that  the  quantity  being  measured  takes  on 
the  value  x  or  less.  Figure  2-3  actually  plots  one  minus  that  proba¬ 
bility,  i.  e.  ,  the  probability  that  the  quantity  is  x  or  greater. 


analysts.  A  better  approach  would  be  cc  3pe.  *  a  ratio  at  some  level 
of  probability  such  as,  say,  90  percent,  i.  c.  ,  wt.  desire  an  effective¬ 
ness  of  0.  90.  Since  we  want  a  performance  level  the  system  will 
achieve  at  least  90  percent  of  the  time,  we  pick  off  the  0.  90  points  on 
the  curves  in  Figure  2-3,  which  are  as  follows  for  each  ’  system:" 

A  =  5,  B  -  66,  C  =  136.  Thus,  the  "90  percent"  performance- 
requirement  ratios  are:  A  =  5/150  =  0.  333;  B  =  66/150  -  0.440; 

C  =  136/150  =  0.907.  This  makes  more  sense  as  a  measure  of  "extent" 
if  in  addition  to  the  150  ton-mile/day  requirement  we  also  require  that 
effectiveness  be  at  least  0.  90.  For  instance,  the  ratio  for  case  B 
indicates  that  we  have  met  44  percent  of  the  required  150  tor-miles/ 
day  at  a  probability  of  0.  90.  If  the  requirement  had  been  only  66  ton- 
miles /day,  we  would  have  an  effectiveness  of  0.  90  and  a  performance- 
requirement  ratio  of  1.0. 

As  a  general  procedure  for  systems  in  which  fractional  per¬ 
formance  may  have  some  meaning,  effectiveness  numbers  will  be 
more- readily  interpreted  if  supplemented  by  a  performance-require¬ 
ment  ratio  at  a  probability  which  is  equal  to  the  minimum  required 
probability  of  success  (effectiveness). 

In  fact,  the  effectiveness  and  the  performance- requirement  ratio 
are  simply  two  ways  of  looking  at  the  same  data.  Effectives  is  a 
probability  associated  with  a  "fixed"  performance  whereas  l  , 
performance -requirement  ratio  (as  defined)  associates  performance 
with  a  "fixed"  probability.  Both  require  the  distribution  of  performance. 

2.2.2  The  Effectiveness  Model 

By  now  it  should  be  clear  that  system  effectiveness  is  a  measure 
of  the  system's  net  usable  performance  in  relation  to  the  type  and 
amount  of  performance  required  by  its  mission.  Not  only  is  the 
system's  designed  or  theoretical  performance  capability  taken  into 
account,  but  al3o  the  many  operational  conditions  and  system  attributes 
which  tend  to  degrade  that  performance  --  including  environmental 
and  operational  influences,  inherent  tendencies  to  fail  or  degrade  with 
time,  etc.  --  together  with  the  resources  for  restoring  the  system  or 
preventing  such  degradation.  The  predicted  or  measured  effectiveness, 
then,  is  the  probability  of  successfully  performing  a  mission  given  the 
net  effect  of  all  the  positive  and  negative  influences  on  performance  in 
an  operational  environment. 
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The  proper  accounting  of  these  factors  requires  the  formulation 
of  an  analytical  framework  or  mathematical  model.  This  is  called 
the  "effectiveness  model"  and  it*  formulation  requires  specification 
of  the  details  of  system  characteristics,  mission  profiles  and  func¬ 
tional  requirements,  planned  support  systems  and  policies,  projected 
demand  rates,  and  use  doctrine. 


The  structure  of  such  models  must  always  be  specific  to  the 
particular  system/mission  under  consideration.  However,  the  con¬ 
ceptual  framework  within  which  the  models  are  structured  tends  to 
be  fairly  consistent  across  broad  classes  of  systems. 

The  general  notion  of  an  affectiveness  model  assumes  that  threat 
analysis  and  analysis  cf  mission  profiles  provide  a  time-oriented 
specification  of  system  functions  to  be  performed  together  with  critical 
performance  levels  and  durations  along  each  of  the  performance  dimen¬ 
sions.  It  is  further  assumed  that  analysis  of  the  system's  character¬ 
istics  in  relation  to  the  mission  situation  provides  a  specification  of 
its  performance,  including  possibilities  of  failure  or  degradation,  with 
respect  to  the  performance  dimensions  associated  with  its  functions. 
Thus,  both  demands  made  by  the  mission  on  the  system's  performance 
and  the  system's  (net  or  usable)  performance  capability  in  response  to 
demands  are  defined  in  such  a  way  that  they  can  be  compared. 

Generally  speaking,  both  mission  demands  and  system  perform¬ 
ance  will  be  variable  due  to  the  statistical  nature  of  (1)  threats  or  other 
factors  which  define  mission  requirements,  and  (2)  system  processes 
which  define  the  presence  or  absence  of  various  levels  of  performance. 
As  a  result,  effectiveness  of  the  ystem  in  meeting  a  particular  type 
of  demand  depends  on  the  probability  tnat  the  demand  on  performance 
is  at  a  given  level,  and  on  the  probability  that  the  performance  of  the 
system  is  at  least  equal  to  or  greater  than  the  given  level  of  demand. 

This  is  illustrated  graphically  in  Figure  2-4,  v/here  curve  (a)  is 
the  probability  density  of  demands,  i.e.  ,  the  probability  that  de  nand 
on  performance  is  x,  and  curve  (b)  is  the  probability  that  usable  sys¬ 
tem  performance  is  x  or  greater.  For  instance,  an  underwater  target 
might  be  at  a  range  of  5  miles  (the  demand)  30  percent  of  the  time,  and 
a  given  sonar  system  might,  after  discounting  environmental  factors, 
"inherent"  failure  and  degradation  possibilities,  operator  problems  and 
the  like,  detect  and  classify  a  target  at  5  miles  or  less  80  percent 
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of  the  time.  According  to  curve  (a,,  target  distances  (demands)  are 
conveniently  defined  at  three  levels  as  shown  in  column  (1)  of  the  table 
(c)  in  Figure  2,-4  with  the  associated  probabilities  given  in  column  {?.). 
System  performance,  as  given  in  curve  (b),  can  meet  these  demands 
or  better  with  the  probabilities  given  in  column  (3).  The  product  of 
probabilities  in  each  row  gives  the  joint  probability,  in  column  (4), 
that  the  demand  is  x  and  that  the  performance  is  x.  or  greater.  The 
sum  of  the  joint  probabilities  in  column  (4),  in  this  case,  is  0.46, 
which  is  the  effectiveness  or  probability  of  success  assuming  indepen¬ 
dence  of  demands  and  performance. 


In  curve  (a)  of  Figure  2-4,  demand  levels  are  represented  as 
discrete  for  simplicity  of  illustration.  In  practice,  demand  densities 
and  performance  level  distributions  would  frequently  be  continuous. 

The  equi/alent  of  the  above  operation  in  continuous  form  is  stated 
mathematically  in  Equation  2-3  given  in  Figure  2-5,  which  expresses 
system  effectiveness  as  the  a  priori  probability  that  a  system  will  be 
;ole  to  meet  any  level  of  mission  demand  chosen  at  random  from  the 
total  distribution  (density)  of  demand  possibilities.  Unavailability  of 
performance  due  to  failure,  damage  or  other  factors  is  included  since 
performance  can  be  zero.  The  equation  can  be  expanded  tc  include 
any  number  of  functions  and  performance  dimensions.  Different  effec¬ 
tiveness  formulations  can  essentially  be  regarded  as  special  cases  of 
this  equation  in  its  continuous  or  discrete  form. 

Equation  2-4  in  Figure  2-5  extends  the  concept  to  missions  in 
which  partially  meeting  a  demand  has  some  value  or  utility.  This  is 
particularly  important  where  degradation  as  opposed  to  total  failure 
may  occur.  For  instance,  target  damage  may  have  some  value  even 
though  the  objective  is  total  "kill,  "  communication  system  degradation 
may  affect  speech  intelligibility  rather  than  totally  blocking  trans¬ 
mission,  etc. 


A  frequently  encountered  special  case  occurs  when  the  distribution 
of  demands  has  not  been  derived  and  the  requirement  on  level  of  per¬ 
formance  is  specified  as  a  fixed  number;  in  that  case  no  credit  is 
given  for  degraded  states  which  have  some  probability  of  meeting 
lower-level  demands.  Effectiveness  is  underestimated  in  such  cases. 


The  foregoing  is  a  very  general  way  of  looking  at  the  effective¬ 
ness  concept.  It  provides  a  modeling  starting  point  which  can  encom¬ 
pass  a  wide  variety  of  systems,  missions  and  criteria.  However, 
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typical  formulations  have  started  at  a  somewhat  less-general  level 
through  the  use  of  particular  classes  of  conditional  probabilities  (effec¬ 
tiveness  "elements")  which  structure  the  modeling  process.  These 
probabilities  are  related  to  a  generalized  way  of  looking  at  a  large 
class  of  Navy  missions;  they  can  also  be  readily  related  to  typical 
system  acquisition  program  elements  and  functions  --  hence  their 
practical  utility.  The  next  section  presents  this  commonly-used 
contemporary  approach  to  effectiveness  modeling  with  the  understanding 
that  other  approaches  can  be  used  and,  for  some  classes  of  problem, 
maybe  preferable  or  even  necessary. 

2.2.2.  1  Major  Elements  of  a  System  Effectiveness  Model 

In  Figure  2-1  system  effectiveness  is  represented  as  a  function 
of  performance  capability  factors  and  time-relate  ’.  factors.  Aside 
from  special  cases,  a  system  may  be  thought  of  as  being  required  at 
some  point  in  calendar  time  (which  can  be  regarded  as  random  for 
many  Navy  defense  systems),  after  which  it  must  dependably  perform 
a  mission  for  a  specified  period  of  time  (mission -time)  given  its 
designed  performance  capabilities.  The  effectiveness  of  a  system, 
then,  depends  on  its  availability,  dependability,  and  capability  in  rela¬ 
tion  to  the  mission.  The  following  definition  of  system  effectiveness 
is  given  by  MIL -STD- 72 IB  (which  is  similar  to  one  provided  by  the 
Weapons  System  Effectiveness  Industry  Advisory  Committee 
[  WSEIAC]  *): 

SYSTEM  EFFECTIVENESS:  A  measure  of  the  degree  to  which 
an  item  can  be  expected  to  achieve  a  set  of  specific  mission 
requirements,  and  which  may  be  expressed  as  a  function  of 
availability,  dependability  and  capability.  ** 


Weapons  System  Effectiveness  Industry  Advisory  Committee 
(WSEIAC),  Final  Reports  AFSC-TR-65- 1 ,  2,  3,  4,  5,  6, 

January  1  965 . 

MIL-STD-721B  "Definitions  of  Effectiveness  Terms  for  Reliability, 
Maintainability,  Human  Factors,  and  Safety"  25  August  1966. 


This  definition  may  be  expressed  as 


Eg  =  f(A,  D,  C)  (  «  A'  [D]  C  in  the  WSEIAC  approach) 

where,  according  to  MIL-STD-721B: 

A  =  AVAILABILITY:  A  measure  of  the  degree  to  which  an  item 
is  in  the  operable  and  committable  state  at  the  start  of  the 
mission,  when  the  mission  is  called  for  at  an  unknown 
(random)  point  in  time 

D  =  DEPENDABILITY:  A  measure  of  the  item  operating  con¬ 
dition  at  one  or  more  points  during  the  mission,  including 
the  effects  of  Reliability,  Maintainability  and  Survivability, 
given  the  item  conditions)  at  the  start  of  the  mission.  It 
may  be  stated  as  the  probability  that  an  item  viill  (a)  enter 
or  occupy  any  one  of  its  required  operational  modes  during 
a  specified  mission,  (b)  perform  the  functions  associated 
with  those  operational  modes. 

C  =  CAPABILITY:  A  measure  of  the  ability  of  an  item  to 
achieve  mission  objectives  given  the  conditions  during 
the  mission. 

These  are  usually  expressed  as  probabilities.  For  instance,  in  the 
WSEIAC  approach: 

A  is  the  Availability  vector  (a  vector  array  of  various  state 
probabilities  of  the  system  at  the  beginning  of  the  mission), 
and  A'  is  its  transpose 

[D]  is  the  Dependability  matrix  (a  matrix  of  conditional  proba¬ 
bilities  over  a  time  interval,  conditional  on  the  effective  state 
of  the  system  during  the  previous  time  interval) 

C  is  the  Capability  vector. 

The  three  major  elements  of  system  effectiveness  together  with 
some  of  the  more-important  factors  which  influence  them  are  shown 
diagrammatically  in  Figure  2-6. 


-20- 


— -----  ..... «... 


SYSTEM  EFFECTIVENESS 


Some  factors  affecting  system  effectiveness 


Although  it  may  not  be  entirely  clear  from  the  definitions, 
the  three  terms  are  mutually  exclusive:  great  care  must  be  exer¬ 
cised  in  modeling  to  guarantee  that  the  same  data  are  not  included  in 
more  than  one  term.  Further  explanation  may  help  to  clarify  the  terms. 

2.2.2.  1.1  Availability 

Availability,  as  ordinarily  defined,  is  simply  the  probability  that 
the  system  is  in  an  "up"  and  ready  state  at  the  beginning  of  the  mission 
when  ti  e  mission  occurs  at  a  random  point  in  time,  i.  e.  ,  is  ready  to 
operate  within  allowable  response  time  with  ail  mission- required 
functions  capable  of  operating  within  design  specifications.  Availability 
(of  hardware)  is  a  function  of  the  reliability  and  maintainability  charac¬ 
teristics  of  the  system  (neglecting,  for  the  moment,  safety,  logistics, 
human  facti  rs,  and  vulnerability/ survivability  properties  of  systems). 
For  sufficiently  long  operating  periods,  \ vailability  of  a  system  (with 
zero  warning  time)  can  be  expressed  in  t  rms  of  its  MTBF  (mean  time 
between  failures)  and  its  MTTR  (mean  time  to  restore  or  mean  active 
repair  time)  given  that  it  has  failed: 

_ MTBF _ 

1  =  ~MTBF  +  MTTR 

where  the  subscrip1  J_ indicates  that  'it's  is  "Inherent"  Availability,  i.  e.  , 
ideal  availability  as  mining  only  acti  n  components  of  corrective  main¬ 
tenance  time,  i.e.  ,  io  waiting  for  :  ;s  and  technicians;  no  "detec¬ 
tion"  or  "administrat've"  time;  no  dov  -.time  due  to  preventive  main¬ 
tenance  or  servicing;  immediate  availability  of  technical  manuals,  test 
equipment,  and  software;  etc. 

If,  in  addition  to  c  orrective  n-.a  .tenance,  preventive  maintenance 
is  also  included,  the  computed  availability  is  commonly  called  "Achieved 
Availability:" 

_ v.rBM_ 

a  =  MTBM  +  MADT 


where  MTBM  is  mean  time  between  maintenance  (both  corrective  and 
preventive)  and  MADT  is  mean  active  downtime,  which  includes  the 
active  (non-waiting)  time  elements  of  both  preventive  and  corrective 
maintenance. 

Both  Aj  and  A&  are  frequently  found  in  specifications  imposed  on 
the  producer  or  contractor  where  the  producer  or  contractor  is  not 
held  responsible  for  extra-system  logistic  and  other  user  factors  beyond 
his  control.  While  "Achieved  Availability"  comes  closer  than  "Inherent 
Availability"  to  representing  the  state  of  affairs  in  an  operational  situa¬ 
tion,  there  are  still  significant  discrepancies  between  so-called 
"Achieved"  and  Operational  Availabilities.  The  measure  of  ultimate 
concern  to  the  user,  and  thus  of  greatest  relevance  in  system  effec¬ 
tiveness  analyses,  is  Operational  Availability. 

In  an  operational  situation,  all  of  the  sources  of  non-operable 
time,  active  and  inactive,  including  "software"  downtime,  supply  and 
administrative  delay  times,  corrective  and  preventive  maintenance 
times,  are  combined  into  the  term  "downtime;"  "Operational"  Hardware 
Availability  (Aq^),  then,  is  a  function  of  MTBM  and  mean  downtime 
(MDT): 


A  _  MTBM 

oh  "  MTBM  +  MDT  ’ 


^oh  *s  a  probability  which  assumes  availability  of  an  operator  if  one  is 
required.  In  man-machine  systems,  Availability  (AQp)  of  the  neces¬ 
sary  operating  personnel  (in  "up"  condition)  must  also  be  determined. 
Operational  System  Availability,  A0,  then,  is: 

A  =  A  A  ,  . 
o  op  oh 

This  measure  of  availability  accounts  for  the  effects  cf  logistics,  main¬ 
tenance  policies,  manning,  priorities,  etc.,  and  is  therefore  the  mea¬ 
sure  of  choice  in  a  system  effectiveness  analysis  and  of  greatest  interest 
to  the  user.  Use  of  Aj  to  compute  Eg,  on  the  other  hand,  would  pro¬ 
vide  something  like  an  "inherent  system  effectiveness,"  which  could 
lose  much  of  the  point  of  doing  such  an  analysis.  A  considerable  toss 


of  analytic  and  design-decision  power  would  be  lost  since  logistic 
problems  (to  mention  only  one  factor!  provide  one  of  the  major  sources 
of  loss  of  effectiveness  in  Navy  systems  in  the  fleet.  In  one  case  of 
a  major  sonar  system,  for  instance.  Inherent  vs.  average  measured 
Operational  Availability  of  seven  ships  was  0.  960  as  compared  to  0.320 
Similar  examples  abound  in  most  if  not  ill  types  of  systems. 

As  mentioned  above,  Inherent  and  Achieved  Availability  are  fre¬ 
quently  used  in  hardware  specifications  wnere  the  contractor  is  not 
held  responsible  for  logistics  and  the  operational  environment  (as  he 
frequently  cannot,  particularly  if  he  is  a  lower-tier  contractor). 
Nevertheless,  system  effectiveness  studies  performed  by  the  Navy 
(particularly  the  user)  for  the  purpose  of  generating  requirements,  or 
by  major  system  contractors,  should  use  Operational  Availability. 
Except  for  very  early  gross  estimates,  this  generally  requires  a  more- 
complex  model  structure,  but  the  payoff  in  ope~  ational  effectiveness 
fully  warrants  the  additional  effort. 

A  factor  which  is  sometimes  overlooked  in  Availability  formula¬ 
tions  is  Operator  Availability  in  a  man-machine  system.  Thus,  the 
formulation  of  Aop  may  be  a  critical  determiner  or  Operational  Avail¬ 
ability,  A0,  since  the  probability  of  the  operator  being  in  an  "up" 
state  (within  allowable  response  time)  may  not  be  particularly  high. 

This  is  a  function  of  manning  policies,  work  duty  cycles,  training, 
safety,  health  factors ,  administrative  procedures,  personnel  vulner¬ 
ability  in  combat  situations,  location  of  duty  areas  and  living  quarters 
in  relation  to  systems,  personnel  cross -utilization  among  systems  or 
functions,  and  numerous  other  factors  which  determine  the  availability 
of  adequately- trained  operators  at  the  system  site  within  allowable 
response  times. 


In  addition  to  Operator  Availability,  Maintenance  Technician 
Availability  has  an  impact  on  the  mean -downtime  (MDT)  term  of 
Hardware  Availability  since  delays  due  to  "waiting  for  technician"  are 
included.  Again,  manning  policies  and  priorities  plus  demand  rate 
(determined  by  the  MTBM  terms  of  all  the  equipments /systems  he 
services)  determine  technician  waiting  time. 


Estimating  these  Availabilities,  except  in  the  simplest  cases, 
generally  requires  that  a  mathematical  model  be  constructed  and  exer¬ 
cised  which  accounts  for  the  interrelation  of  the  many  factors  across 
the  total  system.  Total  ship  (base,  etc.)  models  implemented  by  com¬ 
puter,  utilizing  simulation  or  equivalent  techniques,  are  frequently  ibe 
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only  feasible  means  of  doing  this.  Such  models  not  only  estimate 
Availability  for  given  support  conditions,  but  also  provide  a  means 
for  determining  optimal  manning,  logistics,  maintenance  scheduling 
and  related  policies. 

2.2.2. 1.2  Dependability 

Dependability  is  the  probability  that,  given  the  system  was  avail¬ 
able,  it  will  continue  to  operate  throughout  the  misaion  either  (1)  with¬ 
out  a  system-level  failure,  or  (2)  if  it  fails,  it  will  be  restored  to 
operation  within  some  critical  time-interval  which,  if  exceeded,  would 
result  in  mission  failure.  “Without  a  system-level  failure"  means  that 
the  output  of  the  system  in  response  to  demands  during  the  mission  is 
within  design  specifications,  considering  the  effects  of  the  environment. 
For  instance,  failure  of  a  redundant  item  when  only  one  of  two  is  needed 
to  produce  the  system  output  would  not  imply  system  failure;  faiiure  of 
both,  however,  would  produce  system  faiture.  Various  Dependability 
cases  exist  depending  on  the  criteria  for  system  failure,  and  each 
requires  a  different  model.  Examples  of  such  criteria  are:  (1)  no 
failure  allowable  (reliability  of  serial  systems);  (2)  no  system-level 
failures  allowable,  but  certain  element  failures  may  occur  n-j  times 
without  repair  (reliability  of  systems  where  j_  out  of  ^parallel  elements 
must  function):  (3)  no  system-level  failures  allowable,  but  certain 
element  failures  may  occur  with  repair  a  given  number  of  times  (relia¬ 
bility  with  repair  of  parallel  system  elements  where  number  of  spares 
and  number  of  technicians  are  specified);  (4)  system-level  failures 
allowable  if  downtime  is  less  than  a  specified  time;  (5)  other  cases  also 
exist.  (The  first  three  of  these  cases,  together  with  Availability,  are 
within  the  capability  of  the  Generalized  Effectiveness  Methodology 
(GEM),  a  Navy -developed  computerized  model  which  is  discussed  in 
Appendix  B). 

As  in  the  case  of  Availability,  Dependability  is  also  a  function  of 
the  reliability"  and  maintainability  characteristics  of  a  f'Stem  (not  to 
mention  vulnerability/survivability,  safety  and  others).  However,  as 
mentioned  above,  many  models  exist  depending  on  the  type  of  system/ 
mission  and  the  criteria  for  failure.  The  simplest  one  (no  failures  in 
a  simple,  serial  system)  eliminates  the  maintainability  term  and  is 


Reliability  should  include  not  only  "inherent"  failures,  but  failures 
induced  by  maintenance  actions  as  well. 


simply  the  equation  for  system  reliability  given  in  Figure  2-7,  Equa¬ 
tion  2-5.  If  a  system-downtime-per-failure  of  time  jt_or  less  means 
that  the  mission  is  not  failed  but  can  continue  after  repair  of  the  sys¬ 
tem.  then  under  certain  assumptions  an  equation  such  as  Equation  2-6 
of  Figure  2-7  is  an  appropriate  expression  for  Dependability.  If  the 
probability  of  having  more  than  one  failure  is  negligible,  an  approxima¬ 
tion  which  has  been  used  frequently  is  given  by  Equation  2-7. 

The  equations  in  Figure  2-7  are  given  as  illustrations  only*  since 
operational  situations  are  not  generally  as  simple  as  this.  For  instance, 
assumptions  such  as  exponentially  distributed  downtime  usually  do  not 
hold  except  as  convenient,  but  usually  inaccurate,  approximations. 

(The  exponential  distribution  is  more  likely  to  be  a  reasonable  approxi¬ 
mation  for  inherent  repair  times  than  for  downtime.)  Furthermore, 
human  performance  terms  are  not  included  (except  as  implied  in  MDT) 
and  sources  of  operator  "failure"  and  system  downtime  due  to  operator 
"outage"  must  be  considered. 

As  in  the  case  of  Availability,  Dependability  of  a  man-machine 
system  must  obviously  include  the  effect  of  Operator  Dependability  in 
addition  to  Hardware  and  Software  Dependability.  Careful  examination 
of  a  mission  profile  and  the  functional  requirements  may  reveal  many 
potential  sources  of  functional  failure  and  downtime  due  to  operator 
actions  or  absence  of  actions.  These  may  be  in  either  of  two  forms: 
human  induced  hardware  failure  due  to  operator  "goofs;"  or  failure  to 
perform  operator  functions  due  to  conflicting  duties,  lack  of  alertness, 
removal  from  duty  by  sickness,  accident  or  damage,  error,  etc. 

Many  of  these  are  similar  to  those  which  affect  Availability,  except 
that  they  occur  during  the  mission  rather  than  prior  to  or  at  its  begin¬ 
ning.  In  some  cases  it  will  be  more  convenient  to  include  them  in  a 
combined  hardware-personnel  model;  in  some  cases,  separate  con¬ 
sideration  is  valid.  Where  two  sources  of  "undependability"  are  not 
interdependent,  they  can  be  estimated  separately  and  combined  to 
estimate  Total  Man-Machine  Dependability.  Thus  where  independence 
can  be  assumed,  if  Dh  is  Hardware  Dependability  and  Dp  is  Operator 
Dependability,  then  Overall  System  Dependability  is 

D  =  D.  D  . 
h  p 
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Figure  2-7.  Illustrative  equations  for  simple  cases  of  Hardware  Dependability 


Frequently,  however.  Dependability  of  the  hardware  may  be  affected 
by  Operator  Dependability  (as  well  as  by  Maintainer  Dependability, 
which  must  appear  in  the  hardware  term).  In  that  case,  joint  distri¬ 
butions  of  operator  and  hardware  factors  must  be  developed  in  order 
to  compute  Overall  System  Dependability. 

Care  must  be  exercised,  however,  that  the  same  personal  effec¬ 
tiveness  factors  are  not  included  in  both  the  Dependability  and  Capa¬ 
bility  terms.  To  prevent  this,  it  is  sometimes  better  (where  possible) 
to  incorporate  all  the  operator  effectiveness  factors  into  a  single  con¬ 
ditional  probability  (personnel  subsystem  effectiveness)  and  multiply 
this  times  the  "hardware"  effectiveness  number.  However,  this  is 
often  not  possible  because  substantial  dependencies  may  exist.  An 
example  of  the  dilemma  which  can  arise  is  given  by  the  problem  of 
human  error  (estimated  by  "human  reliability"  practitioners).  This 
could  either  be  treated  as  a  failure  analogous  to  hardware  failure  and 
included  in  the  Dependability  term,  or  as  a  performance  capability 
factor  and  included  in  the  Capability  term.  Although  conceptually  it 
probably  is  treated  more-easily  in  the  Capability  term,  it  usually  makes 
little  difference  where  it  appears  providing  it,  or  terms  implying  it, 
appear  only  once. 

2.2.2.  1.3  Capability 

Capability  is  probably  the  most- misunderstood  of  the  three  effec¬ 
tiveness  terms.  It  is  the  probability  that  the  s/siem's  designed  per¬ 
formance  will  allow  it  to  meet  mission  demands  successfully  assuming 
that  the  system  is  Available  and  Dependable.  This  term  takes  into 
account  the  adequacy  of  the  system  performance  elements  to  carry  out 
the  mission  when  operating  in  accordance  with  the  system  design  speci¬ 
fications  as  affected  by  the  environment.  Both  machine  and  human 
modules  of  the  operable  system  are  included.  Mission  analysis,  includ¬ 
ing  mission  profiles  with  quantitative  demand  levels,  are  required  to 
formulate  a  Capability  model. 

The  definition  of  Capability,  then,  implies  that  when  the  system 
is  carrying  out  a  mission  where  demands  are  within  its  design  per¬ 
formance  limits,  and  under  environmental  conditions  that  are  within 
its  specified  limitations,  it  might  be  assumed  that  its  Capability  (stated 
in  probabilistic  terms  of  mission  success)  is  exactly  what  its  designers 
computed  or  determined  by  test.  This  is  rarely  1.0,  as  is  sometimes 
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assumed  in  effectiveness  computations,  except  in  vastly  over-designed 
systems  or  very  special  cases.  Usually,  however,  Capability  is  even 
less  than  theoretical  computations  or  tes‘  results  would  indicate  since 
demands  or  environment  may  not  be  precisely  within  specified  limits, 
electromagnetic  compatibility  problems  exist,  etc.  ,  plus  the  fact  that 
the  human  performance  part  of  the  Capability  term  may  have  been 
overestimated  or  even  assumed  to  be  1.0  t'~  hich  is  the  assumption  when 
the  human  performance  term  is  effective* /  left  out).  Thus,  the  elfec- 
tiveness  modeler  must  usually  modify  the  system  performance  numbers 
obtained  from  hardware  designers  in  order  to  obtain  a  more-accurate 
estimate  of  Total  System  (man-machine  system)  Capability.  This  is 
frequently  difficult  and  requires  explicit  consideration  of  the  human 
performance  term  in  cooperation  with  human  factors  specialists.  An 
approach  to  "personnel  subsystem  effectiveness"  modeling  should  be 
used  which  provides  outputs  in  a  form  compatible  with  the  other  effec¬ 
tiveness  submodels.  The  form  of  such  models,  at  the  most  general 
level,  is  similar  to  the  form  of  the  general  effectiveness  equations  given 
in  Figure  2-5.  The  major  difficulty  is  in  obtaining  suitable  data;  care¬ 
fully-controlled  estimation  processes  must  frequently  substitute  for 
empirically-obtained  data  due  to  the  budgets  and  schedules  of  typical 
programs. 

Assuming  that  sources  of  operator  "outage"  are  included  in  the 
Dependability  term,  the  (human)  Capability  term,  Cp,  is  the  probability 
that  the  operator  responds  successfully  to  mission  demands  assuming 
(1)  that  he  is  Dependable,  (2)  that  the  hardware  is  Dependable,  and  (3) 
that  the  hardware  is  Capable.  The  term  will  include  human  "error" 
and  will  reflect  the  effects  on  operator  performance  of  training,  experi¬ 
ence,  change  of  performance  as  a  function  of  mission  stress  and  dura¬ 
tion,  motivation  factors,  etc.  Some  of  these  can  be  estimated  from 
experimental  data  or  operational  records,  but  many  are  at  present 
known  only  qualitatively  and  their  effects  must  be  estimated  on  the 
basis  of  judgment.  The  temptation  is  to  leave  them  out.  However, 
their  inclusion  through  the  cooperative  efforts  of  the  modeler  (who 
structures  the  form  of  input),  human  factors  ’pecialists,  and  opera¬ 
tional  personnel,  will  invariably  lead  to  a  more-valid  effectiveness 
analysis  output  --  and  may  provide  the  means  for  identifying  sources 
of  effectiveness -degradation  which  can  be  corrected  at  the  outset 
rather  than  later  at  great  expense. 

If  Hardware  Capability,  C^,  is  the  probability  that  the  hardware 
will  successfully  meet  mission  demands  assuming  it  is  Available  a..d 
Dependable  and  assuming  the  operator  is  Available,  Dependable  and 


Capable,  and  Cp  is  as  defined  above,  then  under  assumptions  of 
itochastic  independence,  Overall  System  Capability,  C,  is 


C  =  C,C  . 
h  p 


2.2.2. 1.4  Utilization 

The  foregoing  formulations  assume  that  a  defined,  specific 
mission  profile  is  being  analyzed,  so  that  the  modeling  and  computation 
of  system  effectiveness  incorporates  ail  utilization  factors  (fractional 
use-time  or  demand-time  of  system  functions).  If  this  is  not  the  case, 
and  computations  of  Dependability  and  Capability  are  referenced  to  total 
mission- time,  then  a  given  "function  time"  may  be  obtained  by  multi¬ 
plying  mission- time  by  the  function's  Fractional  Utilization  Time,  i.  e.  , 
average  of  the  ratio  of  function  use  time  to  mission  time.  Therefore, 
if  the  model  is  defined  such  that  Utilization,  U,  is  not  implied  in  the 
"D"  and  "C"  terms,  then  system  effectiveness  is  defined  in  terms  of 
"system  function  effectiveness,  Ej,  "  i.  e.  ,  Es  =  f(Ej ,  £3.  •  •  •  ,  Ej, 

.  . .  ,  En),  where  there  are  n_ functions  used  during  the  mission,  and 
system  function  effectiveness  of  the  ith  function  is 


E. 

1 


g(A., 


D., 

1 


C., 

1 


U.). 


2.2.2  Multiple  Missions 


Even  if  a  system  is  to  be  used  for  more  than  one  type  of  mission, 
it  is  still  advisable  to  define  system  effectiveness  with  respect  to  each 
mission,  since  this  information  is  of  considerable  value  to  planners. 
However,  comparison  of  competing  system  concepts  will  usually  also 
require  that  an  index  of  value  of  each  concept  be  estimated  in  terms  of 
the  total  mix  of  mission  uses. 

Although  the  formulation  of  "system  effectiveness"  proposed  by 
some  practitioners  is  identical  to  the  index  of  value  as  defined  in 
Section  2.  1,  the  distinction  is  worth  preserving,  i.  e.  ,  the  distinction 
between  indices  of  value,  which  include  considerations  of  military 
worth  of  the  missions  performed  (as  well  as  costs  and  penalties),  and 
effectiveness,  which  is  a  probabilistic  measure  related  to  the  system's 
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success  in  carrying  out  the  mission  whatever  its  worth.  Where  the 
index  of  value  does  not  include  costs  or  penalties,  but  does  contain 
weighting  factors  for  utilization  across  missions  and  military  worth 
of  the  missions  themselves,  the  special  term  "Index  of  System  Effec¬ 
tiveness"  is  proposed  for  standard  Navy  use  to  preserve  the  distinc¬ 
tion  between  a  "success  probability"  and  a  weighted  index  of  success 
probabilities  used  as  a  decision  criterion.  "Index  of  Value"  should  be 
used  where  costs  and  penalties  are  also  incorporated. 

One  way  to  formulate  an  Index  of  System  Effectiveness  (I.  S.  E.  ) 
where  muldple  mission-uses  exist  for  a  single  system  is  as  follows: 
Suppose  a  system  is  or  can  be  used  N^j  times  to  perform  the  ith 
mission  in  the^th  year;  that  military  wlrth  is  Wtj  for  the  ith  mission 
in  the  jth  year;  and  that  the  system  effectiveness  is  E'S{j  with  respect 
to  the  ith  mission  in  the  ith  year.  Then 


1.  S.  E.  =E£  N..W..E 

ij  ij  a.. 
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Equation  2-8 


This  turns  out  to  be  identical  to  the  numerator  of  the  equation  for 
"Index  of  Navy  Defense  Effectiveness,  "  the  index  of  value  given  in 
Figure  2-2  (Eq.  2-2).  Njj  could  also  be  replaced  by  Ujj,  fractional 
utilization,  although  that  would  ncc  as  clearly  take  into  account  the 
possibility  that  "faster"  systems  might  be  able  to  perform  more 
missions,  which  could  be  important  for  some  types  of  missions  and 
systems  (e.  g. ,  transportation  systems,  search  systems  under  certain 
demand  conditions,  etc.). 

Z .  3  Special  Tools  and  Techniques 

A  large  variety  of  mathematical  models,  computer  programs, 
methods  and  procedures  have  leen  formulated  for  System  Effectiveness/ 
System  Value  (SE/SV)  applications  _*nd  new  ones  appear  every  year. 

Any  comprehensive  review  of  chem  would  be  outside  the  scope  of  this 
manual.  However,  .e  next  two  sections  and  the  Appendices  summarize 
some  of  the  more  important  tools  and  techniques  with  which  the  manager 
should  be  familiar.  In  particular,  a  general  understanding  of  approaches 
to  modeling  and  estimating  distributions  is  important  (Appendix  A). 

Also,  familiarity  with  some  of  the  Navy  models /methods  (or  sources) 


which  are  available  for  modeling  and  predicting,  evaluating,  trade-off 
and  design  disclosure,  will  be  useful  (Appendix  B).  A  summary  of 
useful  computer  programmed  models  is  given  in  the  Handbook  of 
Systems  Effectiveness  Models.*  Risk  and  Uncertainty,  considered  in 
Section  2.  3.  2,  is  a  particularly  important  conceptual  tool  for  the 
manager. 

2.3.1  Modeling  Techniques 

Numerous  approaches  to  mathematical  modeling  of  mission/ 
system  processes  exist  and  may  be  required  depending  on  particulars 
of  the  modeling  problem.  Reviewing  them  in  any  meaningful  depth 
would  require  a  substantial  reference  work  in  its  own  right  and  is 
clearly  outside  the  scope  of  this  manual.  However,  a  few  general 
observations  concerning  the  nature  of  such  models  together  with  a 
simple  illustration  (in  Appendix  A)  will  help  to  give  the  manager  some 
insight  into  their  formulation. 

Since  the  processes  represented  by  SE/SV  models  are  rarely  if 
ever  deterministic  in  nature  (i.e.  ,  do  not  have  exactly  predictable 
outcomes),  the  basic  modeling  requirement  is  to  represent  the  proba¬ 
bilistic  or  stochastic  nature  cf  the  processes.  This  is  achieved  by 
representing  them  as  "finite  random  processes"  which  account  for  the 
alternative  possibilities  and  variability  that  exist. 

In  general,  the  existence  of  some  system  state  will  lead  to  one 
or  more  possible  events  (actions,  tasks,  etc.)  whose  probabilities  of 
occurrence  depend  on  internal  and/or  external  system  factors.  Each 
possible  event  or  sequence  of  events  leads  to  a  new  system  state  which, 
in  turn,  implies  one  or  mere  alternative  events,  and  so  on.  This  is 
generally  represented  graphically  as  a  network  or  branching  tree  dia¬ 
gram.  The  branches  at  each  juncture  have  associated  conditional 
ranch  probabilities  and  each  event(s)  transitioning  tc  subsequent  states 
as  associated  probability  distributions  of  variables  related  to  effec¬ 
tiveness,  cost,  etc.  A  graphic  model  of  the  process  (network  or  tree 
diagram)  is  used  as  the  basis  for.  and  guides  formulation  of,  the 


Naval  Electronics  Laboratory  Center,  Handbook  of  Systems  Effec¬ 
tiveness  Models,  Technical  Document  187,  30  June  1972.  (For 
availability  to  qualified  users,  contact  NELC,  Code  4100,  San  Diego, 
California.  ) 

- 


\ 

? 


1 


mathematical  model.  A  simple  illustration  is  given  in  Appendix  A. 
Appendix  B  summarizes  three  proceduralized  methods  or  models  which 
the  manager  may  find  useful:  the  Generalized  Maintainability  Method 
(GMM),  the  Generalized  Effectiveness  Method  (GEM),  and  Design 
Disclosure  for  Systems  and  Equipment  (DDSE). 

2,  3.  2  Risk  and  Uncertainty 

In  any  rational  approach  to  decision-making,  evaluation  of  risk 
and  uncertainty  is  necessary  and  may  have  considerable  impact  on 
planning- decisions .  "Risk"  is  used  to  refer  to  the  probability  of 
undesirable  outcomes  based  on  analysis  of  objective  data.  Thus,  the 
risK  of  running  out  of  spares  during  a  mission  can  be  computed  from 
reliability  data,  and  can  be  reduced  to  any  desired  level  by  improving 
reliability,  increasing  the  number  of  spares,  providing  on-board 
repair  capability,  or  seme  con.oination  of  these  approaches.  Similarly, 
the  risk  of  repair- lime  exceeding  some  given  value  can  be  determined 
from  the  repair-time  distribution. 

"Uncertainty"  is  generally  used  to  refer  to  subjective  estimates 
of  the  probability  of  undesirable  outcomes.  For  instance,  the  proba¬ 
bility  of  cost  or  schedule  overrun  must  frequently  be  based  on  subjec¬ 
tive  estimates  of  cost  and  time  distributions.  In  either  case,  however, 
the  analytic  procedures  can  be  similar,  and  for  decision-making 
purposes,  both  risk  and  uncertainty  may  need  to  be  combined  in  a 
single  analysis.  In  such  cases,  an  "estimated  risk"  is  derhed. 

Generally  speaking,  in  any  process  3ubj.®ct  to  variability,  at 
least  two  types  of  risk  or  uncertainty  can  be  identified.  First,  there 
is  risk  cr  uncertainty  which  has  its  source  in  "real-world"  variability. 
Thus,  the  lime  to  repair  a  system  failure  may  vary  from  one  repair 
to  another  due  to  differences  in  skill  level  of  maintenance  technicians, 
varying  motivations  and  mission  stresses  for  a  given  technician,  type 
of  failure,  differences  in  surrounding  circumstances  and  lime-pressures, 
etc.  ,  so  that  the  exact  repair- time  is  never  predictable.  However,  the 
distribution  of  time  can  be  defined  by  observing  many  repairs  such  that 
the  probability  can  be  stated  that  repair -time  will  not  exceed  any  given 
value.  This  accounts  for  "real-world"  variability  since  it  recognizes 
the  fact  of  the  underlying  variability  of  the  process. 
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The  second  source  of  variability  arises  from  our  limited  ability 
to  estimate  the  "real-world"  distribution  of  a  variable  or  parameter. 
For  instance,  if  we  had  estimated  a  repair-time  distribution  by  taking 
five  observations  of  repair -time  and  fitting  a  distribution  curve  to  the 
points,  we  would  have  an  "estimate"  of  the  distribution,  but  ten  points 
would  have  given  a  better  estimate  and  50  points  an  even  better  one. 
Thus,  there  is  an  inherent  risk  in  the  estimate  of  any  point  on  the  dis¬ 
tribution  or  any  of  its  parameters  such  as  the  mean  or  standard  devia¬ 
tion,  due  to  the  fact  that  the  distribution  is  based  on  a  "sample"  rather 
than  all  of  the  repair- times  (which  could  never  be  achieved).  This  is 
expressed  by  the  "sampling  distribution"  associated  with  a  parameter 
or  percentile  of  a  distribution,  or  by  a  "confidence  interval"  derived 
from  the  sampling  distribution.  A  confidence  interval  on  the  mean, 
for  instance,  is  an  interval  within  which  a  sample  mean  could  be 
expected  to  fall  with  some  stated  probability.  It  can  also  be  thought  of 
as  an  interval  which  has  a  stated  probability  of  containing  the  true  or 
copulation  mean.  Obviously,  the  wider  the  interval  the  less  certain 
we  are  of  the  true  value  of  the  mean.  Thus,  we  would  expect  the  width 
of  the  confidence  interval  to  decrease  as  the  number  of  observations 
increases.  In  fact,  given  the  number  of  observations  and  the  data  or 
form  of  distribution,  a  sampling  distribution  and  associateu  confidence 
interval  can  be  calculated  or  approximated  for  any  property  of  a  dis¬ 
tribution,  including  the  percentiles  as  well  as  the  mean. 

The  relation  between  the  distribution  of  a  variate  such  as  repair¬ 
time,  accuracy,  etc.  ,  and  the  sampling  distribution  of  some  property 
of  the  distribution  is  illustrated  by  Figure  2-8,  where  a  repair-time 
distribution  (from  Appendix  A)  is  shown  with  a  hypothetical  sampling 
distribution  of  the  mean.  The  horizontal  bar  defines  the  symmetric 
90  percent  confidence  interval  (from  the  5th  to  the  95th  percentiles  of 
the  sampling  distribution).  While  curve  A  tells  us  that  there  is  a 
90  percent  probability  that  repair-time  is  1. 05  ho.irr-  or  less  and  the 
estimated  mean  repair  time  is  0.50  hours,  curve  B,  the  sampling 
distribution  of  the  mean,  tells  us  that  due  to  the  sample  size  used  to 
estimate  the  distribution  (curve  A),  there  is  a  90  percent  chance  that 
a  sample  mean  will  lie  somewhere  between  0.43  and  0.  57  hours.  This 
is  equivalent  to  3aying  that  if  a  large  number  cf  samples  of  the  same 
size  were  taken,  90  percent  of  the  sample  means  would  be  expected  to 
fall  within  (lie  interval.  We  could  also  make  such  statements  as  "the 
probability  is  0.95  that  (lie  mean  is  0.57  hours  or  less." 
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8.  Repair  time  distribution  with  sampling  distribution 
of  the  mean 


For  distributions  which  were  estimated  subjectively  rather  than 
on  the  basis  of  objective  sampling  by  observation,  something  roughly 
akin  to  a  sampling  distribution  m?y  be  arrived  at  which,  in  subsequent 
analytic  procedures,  can  be  used  in  a  similar  way.  However,  the  basis 
and  approach  for  this  are  beyond  the  scope  of  this  manual. 

One  type  of  risk  estimation  which  should  accompany  SE/SV 
analyses  is  to  estimate  the  probability  that  effectiveness,  schedule, 
cost  or  system  value  will  be  worse  than  the  predicted  value  by  varying 
amounts.  This  can  be  achieved  by  computing  or  estimating  sampling 
distributions  or  confidence  intervals  on  all  parameters  of  the  model 
and  combining  them,  using  appropriate  statistical  methods,  to  provide 
a  distribution  of  the  overall  output  measure.  For  instance,  in  the 
sample  SE  computation  (Appendix  A),  ."confidence  intervals"  on  each 
of  the  distribution  parameters  (actually,  "sampling"  distributions) 
could  have  been  combined  appropriately  to  produce  either  of  the  esti¬ 
mated  distributions  of  SE  shown  in  Figure  2-9. 

If  we  had  obtained  curve  A,  we  would  have  less  confidence  in  the 
computation  of  SE  than  if  we  had  obtained  curve  B.  For  instance, 
curve  A  indicates  a  probability  of  0.  25  that  the  true  value  of  SE  might 
actually  be  less  than  0.  63,  while  curve  B  indicates  a  probability  of 
only  0.08.  In  both  cases,  the  "best"  estimate  is  0.70  (the  "mean"). 
Curve  A  might  represent  the  low  quality  of  input  data  early  in  the 
acquisition  program,  while  curve  B  might  represent  higher  quality 
data  obtainable  in  later  phases.  Decision  confidence  is  low  for  curve  A 
(we  are  90  percent  sure  that  SE  falls  somewhere  between  0.  54  and  0.  86), 
whereas  decision  confidence  is  higher  for  curve  B  (we  are  90  percent 
sure  that  SE  falls  somewhere  between  0.  62  and  0.  78).  If  we  wanted  to 
use  an  estimate  of  SE  for  which  the  estimated  risk  is  10  percent  that 
the  true  value  could  be  lower,  their  respective  10th  percentiles  would 
force  us  to  assume  that  SE  =  0.  57  for  curve  A  and  SE  =  0.  64  for  curve 
B,  even  though  in  both  cases  the  "best  estimate"  was  0.70. 

A  similar  use  of  distributions  can  be  made  to  assess  risk  in  the 
case  of  cost,  schedule,  downtime  or  any  other  measure  of  interest. 
Where  objective  ("hard")  data  are  not  available,  subjective  estimates 
of  upper,  lower  and  central  values  of  the  range  of  variability  may  be 
used  to  provide  points  on  "subjective"  distributions,  after  which  curve¬ 
fitting  procedures  can  be  used  to  find  the  "best-fit"  distribution.  The 
subjective  distribution  is  then  used  to  evaluate  "estimated"  risk.  In 
such  cases,  it  is  generally  advisable  to  estimate  distributions  at  lower 
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levels  of  indenture  and  combine  them  to  provide  the  overall  distribution. 
For  instance,  if  overall  cost  is  the  sum  of  20  item  costa,  the  cost  esti¬ 
mators  would  provide  high,  low  and  central  ("average")  estimates  to 
represent  cost  uncertainty  for  each  item.  The  analyst  would  then  fit 
distributions  to  each  .set  of  estimates  and  combine  them  by  convolution 
to  estimate  the  overall  cost  distribution.  Management,  then,  would 
have  the  option  of  using  the  "best"  or  "mean"  cost,  or  of  using  a  cost 
associated  with  some  estimated  risk. 

2.4  The  Effectiveness  Disciplines 

In  addition  to  the  traditional  design  engineering  functions  which 
determine  hardware  performance  capabilities,  a  large  number  of 
engineering  specialities  or  disciplines  have  become  established  in 
response  to  the  growth  of  system  engineering  and  system  effectiveness 
analysis  in  modern  engineering  management.  These  are  known  as  the 
"effectiveness  disciplines"  since  they  are  related  in  various  ways  to 
assuring  the  effectiveness  of  the  operational  system  through  control 
of  one  or  mere  factors  which  influence  the  Availability,  Dependability 
or  Capability  of  a  system.  Unfortunately,  the  concept  of  an  integrated 
system  engineering  and  effectiveness  program  arose  relatively  late  in 
the  evolution  of  engineering  management,  and  as  a  result,  considerable 
overlap  and  cross -purposes  may  be  found  in  typical  specialty  engineer¬ 
ing  groups.  This  must  be  guarded  against  by  a  carefully- conceived 
effectiveness  management  plan  implemented  within  the  system  engi¬ 
neering  program. 

The  effectiveness  disciplines  include: 


Reliability 
Maintainability 
Logistic  Support 

Human  Factors  (including  Training,  Human  Reliability, 
Human  Engineering,  Habitability,  Manning,  etc.) 

Safe  ty 

Vulnerability /Survivability 
Electromagnetic  Compatibility 


These  are  the  disciplines  which  are  generally  most  central  to  the 
effectiveness  program.  Numerous  others  have  varying  degrees  of 
relationship,  depending  on  the  type  of  system  and  program,  and  must 
also  be  integrated  into  the  acquisition  program.  They  include  such 


specialties  as  silencing,  system  mass  properties,  operating  doctrine, 
value  engineering,  transportability,  producibility,  security  engineering, 
electronic  warfare,  standardization,  etc. 

How  the  effectiveness  elements  and  disciplines  are  related  to 
system  effectiveness  is  illustrated  in  the  system  state  model  given  in 
Figure  2-10.  The  various  effectiveness  engineering  functions  or 
disciplines  which  have  an  impact  on  each  term  (state  or  event)  are 
noted.  Together,  they  constitute  the  scope  of  the  system  effectiveness 
activity  which  determines  the  effectiveness  of  the  system  and  provides 
inputs  to  the  effectiveness  model.  In  a  well- conceived  engineering 
program,  the  effectiveness  modeling  function  provides  a  basis  for 
integrating  these  disciplines  through  its  central,  coordinating  function 
as  the  analytic  arm  of  the  system  engineering  function.  The  relation¬ 
ship  of  effectiveness  and  its  major  variables  and  factors  is  also  sug¬ 
gested  in  Figure  2-6  given  previously. 

Following  are  brief  descriptions  of  the  more- central  effectiveness 
disciplines  (sometimes  referred  to  as  the  "ilides")  which  can  have  a 
major  impact  on  the  effectiveness  of  systems  through  participation  in 
system  engineering  and  design. 


MIL-STD-721B  defines  Reliability  as  "The  probability  that  an 
item  will  perform  its  intended  function  for  a  specified  interval  under 
stated  conditions." 

The  reliability  function  in  engineering  programs  is  one  of  the 
oldest  and  best-established  of  the  effectiveness  disciplines.  Typical 
program  activities  include  modeling,  prediction,  failure  mode  and 
effects  analysis ,  allocation,  test  and  evaluation,  specification  of 
reliability  design  approachef ,  parts  selection,  design  review,  relia¬ 
bility  research,  and  other  activities  necessary  to  provide  assurance 
that  the  system's  failure  characteristics  are  compatible  with  the 
required  Availability  and  Dependability  of  the  system.  The  relation 
of  the  MTBF  (a  major  reliability  parameter)  to  Availability  and 
Dependability  has  already  been  discussed  in  earlier  sections. 
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Inherent  Maintain- 

Reliability  r-  ability 


Example  top-level  system  effectiveness  diagram  with  related 
disciplines 


Figure  2-10.  (Continued) 


One  of  the  major  tools  of  the  reliability  analyst  is  the  reliability 
block  diagram.  This  is  a  block  diagram  of  the  system  defining  the 
series  and  parallel  structure  of  the  system  elements,  and  forms  the 
graphic  basis  for  the  mathematical  model  (reliability  model)  which 
'  tates  the  probability  of  system  failure  during  missions  of  stated  dura¬ 
tion.  Since  each  element  must  have  a  defined  failure  rate,  prediction 
data  and„methods  are  essential.  Various  compendia  of  such  data  are 
available  and  are  continually  updated  through  industry-  and  service¬ 
wide  laboratory  and  field  data  collection  programs.  System  function 
and  mission  analyses  are  also  essential  to  obtain  utilization  factors 
(function  "on"  time),  and  detailed  failure  modes  and  effects  analyses 
are  further  required  to  establish  the  relation  between  parts  failures 
and  consequences  to  mission  success. 

When  predicted  reliability  is  unsatisfactory,  it  is  the  responsi¬ 
bility  of  the  reliability  function  to  recommend  cost-effective  reliability 
improvement  techniques  which  will  enable  goals  to  be  achieved.  Typi¬ 
cally,  these  include  the  use  of  redundancy,  parts  selection  and  screen  ¬ 
ing,  derating,  cooling  and  special  designs.  It  is  essential  that  this  be 
done  23  part  of  the  overall  effectiveness  program  in  order  to  guarantee 
that  the  relevant  trade-offs  with  other  system  attributes  are  accom¬ 
plished.  In  an  ideal  program,  the  design  function  of  the  reliability 
organization  alternates  with  the  modelir.g/evaluation  function  which  is 
carried  out  in  concert  with  maintainability,  logistics,  human  factors , 
etc.  ,  within  an  overall  system  effectiveness  analysis  function. 

2.4.2  Maintainability  (M) 

MIL-STD-721B  defines  Maintainability  as  "A  characteristic  of 
design  and  installation  which  is  expressed  ?s  the  probability  that  an 
item  will  be  retained  in  or  restored  to  a  specified  condition  within  a 
given  period  of  time,  when  the  maintenance  is  perforn.ed  in  accordance 
with  prescribed  procedures  and  resource?."  This  is  distinguished 
from  maintenance,  v/hich  is  defined  as  "All  actions  necessary  for 
retaining  an  item  in  or  restoring  it  to  a  specified  condition.  " 

Maintainability  is  also  well-established  as  an  effectiveness  dis¬ 
cipline  through  the  relation  of  one  of  its  major  parameters,  MTTR, 
to  Availability  and  Dependability  --  as  discussed  earlier.  In  engi¬ 
neering  programs,  the  maintainability  function  must  work  closely 
with  the  reliability  function  within  an  effectiveness  analysis  framework 
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to  establish  the  optimal  levels  of  system  reliability  and  maintainability, 
e.g.  ,  reliability/maintainability/availahility  (R/M/A)  analysis  is 
carried  out  to  establish  the  most  cost-effective  combination  of  main¬ 
tainability  (MTTR)  and  reliability  (MTBF)  design  features  which  meets 
the  requirements  for  system  Availability  and  Dependability.  The 
opportunity  for  trade-off  between  the  two  functions  is  apparent  in  the 
A  =  MTBF/(MTBF  +  MTTR)  formulation  of  Availability  as  well  as  in 
the  illustrative  equations  for  Dependability  given  in  Figure  2-7. 


Typical  maintainability  activities  include  modeling,  prediction, 
allocation,  test/evaluation,  maintenance  concept  and  M  design  specifica¬ 
tion,  design  review  and  other  activities  necessary  to  provide  assurance 
that  the  system's  maintainability  characteristics  are  consistent  with 
system  effectiveness  goals.  Since  the  repair  time  of  a  system  depends 
on  (1)  failure  characteristics,  (2)  detail  design  features  such  as  pack- 
aging,  labeling,  test  points,  etc.  ,  and  (3)  maintenance  technician 
characteristics,  the  maintainability  function  overlaps  those  of  several 
other  functions  including  reliability,  human  facto-s  and  logistics,  with 
which  close  liaison  must  be  maintained.  In  fact,  it  is  not  uncommon 
to  find  similar  requirements  for  analysis  and  design  criteria  related 
to  the  maintenance  features  of  the  system  in  the  specifications  and 
standards  for  human  factors,  maintainability  and  logistics  support  -- 
all  called  out  for  the  same  acquisition  program. 


As  with  reliability,  the  maintainability  organization  alternates 
between  a  design  function,  and  an  analysis /evaluation  function  which  is 
carried  out  within  the  system  effectiveness  analysis  program.  When 
predictions  indicate  unacceptable  levels  of  maintainability,  it  is  the 
responsibility  of  the  maintainability  design  function  to  recommend 
maintainability  design  approaches  v/hich  will  enable  goals  to  be  met. 

In  addition  to  detail  design  recommendations  involving  test-point  levels^ 
marking/labeling,  packaging,  accessibility  and  various  "human  engi¬ 
neering"  criteria  which  are  well-documented  in  "M  Criteria  handbooks,  " 
trade-offs  and  other  analyses  are  used  to  establish  maintenance  ci  ncepts 
--  generally  in  cooperation  with  or  as  part  of  the  Integrated  Logistics 
Support  function.  The  types  of  maintenance  concept  decisions  made  on 
the  basis  of  modeling  and  analysis  include:  repair  vs.  discard  of  failed 
modules;  location  of  module  or  equipment  repair  (shipboard,  tender, 
depot,  contractor,  etc.);  test  point  location;  type  and  location  of  failure 
detection  devices  and  test  equipment  (manual  vs.  automatic,  special  vs. 
general,  built-in  vs.  portable,  etc.);  technician  numbers  and  skill- 
levels;  special  support  equipment,  tools  and  jigs;  lo  *  tion  and  types  of 
maintenance  data  and  instructions;  level  of  modularization  and  degree 
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of  standardization;  etc.  Since  these  decisions  have  considerable  impact 
on  both  effnctiveness  and  cost,  it  is  essential  that  the  alternative  main¬ 
tenance  concepts  be  evaluated  within  an  overall  cost-effectiveness 
context  through  exercise  of  the  overall  cos-t  and  system  effectiveness 
models  wherever  possible.  At  minimum,  suboptimization  through 
lower-level  trade-off  analyses  is  required.  A  number  of  such  trade¬ 
off  models  have  been  formulated  and  applied  successfully. 

2.4.3  Logistic  Support 

Shortages  of  spares  and  delays  in  obtaining  spares,  technicians, 
tools,  support  equipment,  data  and  other  supporting  elements,  are 
responsible  for  the  major  share  of  system  downtime  in  many  opera¬ 
tional  systems.  Recalling  the  relationship  of  the  mean  downtime 
(MDT)  term  to  Operational  Availability  and  Dependability,  it  can  be 
seen  that  logistic  support  is  a  major  determiner  of  the  realized  or 
operational  system  effectiveness.  Through  maintenance  engineering 
analysis  (MEA)  and  the  development  of  plans  for  maintenance  (PFM) 
including  the  specification  of  spares  levels  and  locations,  maintenance 
manning,  maintenance  policies,  etc.,  the  logistics  support  function 
establishes  the  support  system  design  chrough  use  of  the  effectiveness 
disciplines.  In  most  systems,  effectiveness  is  highly  sensitive  to  the 
support  system  design,  as  is  '•ycle  cost.  Accordingly,  older- 
style  logistic  support  activities,  which  nrcvideti  ,,affer-the-factn  spares 
lists  and  support  equiument  d'  sigrs,  h ace  been  supplanted  by  the  more- 
sophisticated  Integrated  Lo/dsti^s  Support  (ILS)  function. 

ILS,  when  fully  implemented,  is  a  syste  n  engineering  function 
performed  throughout  a.  system's  life  c;''le  which  concerns  itself  net 
only  with  the  support  system  dec.gn,  but  also  with  the  inherent  sup- 
portability  of  the  system  being  supported.  According  to  SECNAVINST 
4000.  29,  "Development  of  Integrated  Logistic  Support  f.?r  Systems  and 
Equipments,”  ILS  requires  the  "Coordinated  and  systematic  planning, 
design,  acquisition,  distribution  and  management  of  the  following  major 
elements  of  logistic  support  as  an  integrated  whole:  (1)  planned  main¬ 
tenance;  (2)  support  personnel;  (3)  technical  data  and  pubs;  (4)  support 
equipment;  (5)  spares  and  repair  parts;  (6)  facilities;  (7)  contract  main¬ 
tenance.  ”  Since  the  concept  of  ILS  is  based  on  the  "systematic  inter¬ 
relation  of  the  actions  required  to  obtain  maximum  material  readiness 
and  optimum  cost  effectiveness,  "  and  involves  the  "optimization  of 
total-life  cost  of  the  system  or  equipment  through  analyses  of  potential 
trade-offs  between  reliability  and  maintainability  requirements,  and 
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alternative  logistic  support  methods,  11  it  readily  assumes  a  major  and 
well-defined  role  within  the  system  engineering  program.  In  accor¬ 
dance  with  Navy  policy  set  forth  by  SECNAVINST  4000.  29,  "Instruc¬ 
tions  issued  by  addressees  pertaining  to  the  major  elements  of  logistic 
support  and  to  related  subjects  such  as  configuration  control,  main¬ 
tenance  engineering,  systems  effectiveness,  maintainability,  and  relia¬ 
bility  shall  recognize  and  support  the  concept  of  integrated  logistic 
stpport.  "  The  ILS  function  is  supported  by  the  system  effectiveness 
analysis  function,  which  provides  the  essential  analytic  framework  for 
making  rational  logistic  support  decisions  within  an  overall  operational 
context. 

2.4.4  Human  Factors  (HF)/Human  Engineering  (HE) 

MIL-STD-721B  defines  Human  Factors  as  "A  body  of  scientific 
facts  about  human  characteristics.  The  term  covers  all  biomedical 
and  psychosocial  considerations;  it  includes,  but  is  not  limited  to, 
principles  and  applications  in  the  areas  of  human  engineering,  personnel 
selection,  training,  iife  support,  job  performance  aids,  and  human  per¬ 
formance  evaluation.  "  Human  Engineering  is  defined  as  "The  area  of 
human  factors  which  applies  scientific  knowledge  to  the  design  of  items 
to  achieve  effective  man-machine  integration  ar.d  utilization.  " 

Except  for  rare  cases.  Navy  systems  are  man-  machine  systems. 
The  role  of  the  human  factor  in  determining  a  system's  Availaoility, 
Dependability  and  Capability  was  discussed  earlier,  and  can  often  be  of 
majo.-  significance  in  a  system's  achievable  effecti-  eness .  Even  so, 
the  weakness  of  many  system  engineering  and  effectiveness  efforts  is 
the  failure  to  integrate  human,  factors  considerations  into  rhe  overall 
effectiveness  formulation.  This  is  because  the  human  facto:-  is  often 
difficult  to  assess  quantitatively  in  a  form  which  is  readily  introduced 
into  a  system  effectiveness  model.  However,  recent  advances  in  human 
performance  quantifi cation  and  modeling  have  improved  this  situation. 

Some  of  the  functions  which  come  under  the  general  heading  of 
Human  Factors  include  human  engineering  requirements  determination, 
man-machine  function  analysis,  task  and  time  line  analysis,  human 
engineering  design  of  equipment,  human  performance  reliability  model¬ 
ing  and  prediction,  workplace  layout,  man-machine  allocation  of  func¬ 
tions,  personnel  training  and  selection,  manning,  habitability,  "per¬ 
sonnel  subsystem"  test  and  evaluation,  and  related  activities  such  as 
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design  of  operator  and  maintenance  manuals  and  procedures,  safety, 
etc. 


2.4.5  Safety 


MIL-STD-721B  defines  Safety  as  "The  conservation  of  human  life 
and  its  effectiveness,  and  the  prevention  of  damage  to  items,  consistent 
with  mission  requirements.  " 

System  safety  analysis  is  carried  out  as  an  integral  part  of  a 
system  effectiveness  program  and  involves  the  systematic  identifica¬ 
tion  of  hazards  which  can  arise  during  normal  operations  or  as  a  result 
of  hardware  or  personnel  failures.  As  indicated  in  Figure  2-6,  unsafe 
conditions  may  affect  both  Operator  and  Hardware  Availability  and 
Dependability.  The  system  safety  analytic  process  starts  with  hazard 
analyses  to  identify  and  evaluate  hazards,  formulates  actions  to  elimi¬ 
nate  or  control  hazards  through  modification  of  system  elements  or 
procedures,  and  feeds  these  actions  into  the  effectiveness  evaluation 
process  to  determine  their  impact. 

One  of  the  tools  of  safety  analysis  is  the  fault-tree.  Starting  with 
accidents  defined  by  a  preliminary  gross  hazard  analysis,  the  analyst 
works  backward  to  diagram  contributory  causes  in  the  form  of  a  tree 
with  causative  pathi  as  branches.  Each  critical  path  points  to  indepen¬ 
dent  causal  elements  called  subevents  for  subsequent  action.  In  order 
to  have  maximum  usefulness  in  a  system  effectiveness  program,  the 
analyses  should  be  quantitative  and  the  probabilities  associated  with 
alternative  possibilities  should  be  defined  together  with  uncertainties 
associated  with  input  data. 

Inputs  to  safety  analysis  include  reliability  failure-rate  data, 
human  reliability  data  (human  error  rates),  failure  modes  and  effects, 
and  other  risk  data  derived  in  a  mission  context.  Thus,  it  can  be  seen 
that  a  close  relationship  must  exist  between  the  safety  function  and 
reliability,  human  factors  and  the  other  effectiveness  disciplines. 

2.4.6  Vulnerability/Survivability 

Vulnerability/Survivability  in  a  system  subject  to  enemy  action 
is  related  to  Availability  and  Dependability  and  probability  of  mission 
success  in  a  way  which  is  analogous  to  reliability  and  maintainability. 
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although  in  a  somewhat  more- complex  way  since  interaction  with  the 
Capability  term  may  exist.  Generally,  Vulnerability/Survivability 
requires  a  "war-game"  type  of  approach  involving  both  system  perform¬ 
ance  and  mission  profiles  including  tactical  threats.  It  may  also  be 
necessary  to  define  Vulnerability  arising  from  inherent  failures  and 
downtime  as  well,  and  some  applications  can  become  complex,  requir¬ 
ing  extensive  simulation.  However,  whether  simple  or  complex,  the 
output  essentially  provides,  under  given  conditions,  the  probability 
that  various  system  functions  are  vulnerable  to  enemy  action  and,  given 
such  vulnerability,  the  probability  that  the  function  will  survive,  i.e.  , 
will  continue  to  be  performed  within  system  specifications  or,  if 
damaged,  will  be  restored  within  some  critical  period. 

One  of  the  more-important  trade-offs  is  between  Vulnerability/ 
Survivability  ?md  performance.  For  instance,  additional  armor  in¬ 
creases.  weight  and  decreases  vehicle  maneuverability  but  may  improve 
survivability  given  a  "hit."  On  the  other  hand,  less  armor  increases 
maneuverability  which  may  decrease  the  probability  of  "hit"  but 
decreases  survivability  if  it  is  hit.  The  objective  is  to  achieve  the 
design  which  balances  these  factors  in  such  a  way  that  the  required 
system  effectiveness  is  achieved  and  system  value  is  maximized. 
Similarly,  centralization  versus  decentralization  has  cost,  perform¬ 
ance  and  reliability  versus  vulnerability  implications. 

Other  problems  which  arise  include  the  optimal  placement  of 
equipments  within  a  ship  or  facility  with  limited  space  when  locations 
vary  in  vulnerability  and  equipments  vary  in  mission- criticality;  the 
optimal  amount  of  self-defense,  including  ECM,  as  opposed  to  other 
capabilities  related  to  the  major  objectives  of  the  system  (e.g.  ,  self- 
defense  systems  may  displace  cargo  capacity  and  have  cost  implica¬ 
tions);  etc.  The  optimal  solution  to  such  problems  is  best  achieved 
through  a  system  effectiveness  and  system  value  formulation  which 
considers  explicitly  the  interrelation  of  all  the  factors.  This  becomes 
clear  when  one  considers  the  potential  effects  on  each  other  of 
vulnerability/survivability,  operators  and  maintenance  personnel 
actions,  equipment  failures  and  downtime,  availability  of  spares, 
function  and  subfunclion  redundancies,  and  numerous  other  factors. 

2. 4. 7  Electromagnetic  Compatibility  {EMC) 

Electromagnetic  Compatibility  has  been  defined  in  Department  of 
Defense  Instruction  3 222.  3  of  July  1967  as  "the  ability  of  communica¬ 
tions -electronic  (C-E)  equipment,  subsystems  and  systems  to  operate 


in  their  intended  environments  without  suffering  or  causing  unacceptable 
degradation  because  of  unintentional  electromagnetic  radiation  of 
response."  Furthermore,  the  instruction  defines  design  compatibility 
as  EMC  achieved  by  the  incorporation  of  engineering  characteristics 
or  features  in  all  electromagnetic  radiating  and  receiving  equipment 
(including  antennas)  in  order  to  eliminate  or  reject  undesired  signals, 
either  self  generated  or  external,  and  enhance  operating  capabilities 
in  the  presence  of  natural  or  man-made  electromagnetic  noise.  The 
instruction  goes  on  to  define  operational  compatibility  as  EMC  achieved 
by  the  application  of  C-E  equipment  flexibility  to  ensure  interference- 
free  operation  in  homogeneous  or  heterogeneous  environments  of  C-E 
equipments.  Operational  compatibility  involves  the  application  of 
sound  frequency  management  and  clear  concepts  and  doctrines  to  maxi¬ 
mize  operational  effectiveness  and  relies  heavily  on  initial  achievement 
of  design  compatibility.  * 

According  to  MJL-E-6051D,  "Electromagnetic  Compatibility 
Requirements,  Systems,"  an  EMC  system  design  program  covers  at 
least  the  following  areas:  (1)  subsystem/equipment  criticality  cate¬ 
gories,  (2)  degradation  criteria,  (3)  interference  and  susceptibility 
control,  (4)  wiring  and  cable,  (5)  electrical  power,  (6)  bonding  and 
grounding,  (7)  lightning  protection,  (8)  static  electricity,  (9)  personnel 
hazards,  (10)  EM  hazards  to  explosives  and  ordnance,  {>  1)  external 
environment,  (12)  suppression  components. 

The  importance  of  EMC  in  system  effectiveness  is  clearly 
apparent.  To  be  properly  related  to  the  other  factors  and  to  determine 
its  influence  on  achievable  system  effectiveness,  quantitative  measures 
are  required.  Generally,  this  requires  that  the  effect  of  electrical/ 
electromagnetic  interference  or  other  effects  on  system  performance 
be  predicted  or  measured  within  a  mission  context,  and  that  the  proba¬ 
bilities  of  total  or  partial  interference  with  system  functions  be  accounted 
for  in  the  computation  of  system  effectiveness.  Depending  on  the  type 
of  effect,  EMC  may  enter  into  the  Availability,  Dependability  or  Capa¬ 
bility  term  --or  frequently  in  all  three.  If  an  EMC  problem  leads  to 
total  or  temporary  failure  of  a  function,  it  can  reasonably  be  accounted 
for  in  the  Availability  or  Dependability  terms  depending  on  v/hen  it 


Caine,  S.  ,  "Electromagnetic  Compatibility  in  Systems  Effectiveness" 
in  Proceedings  of  the  NMC  Fifth  System  Performance  Effectiveness 
Conference,  1969. 
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occurs.  If  performance  degradation  rather  than  total  failure  occurs, 
it  is  probably  easiest  to  account  for  in  the  Capability  terms,  although 
with  some  formulations  (e.  g.  ,  Eq.  2-4)  it  could  be  accounted  for  in 
the  Dependability  term  as  well. 
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CHAPTER  THREE:  SYSTEM  EFFECTIVENESS  IMPLEMENTATION 


Within  a  systematic  propram  of  system/mission  analysis, 
system  engineering,  integrated  logistic  support  analysis,  and  test  and 
evaluation,  the  system  effectiveness  approach  is  the  key  to  rational 
generation  of  system  requirements  and  to  subsequent  definition  and 
development  of  systems.  This  is  due  to  the  following  reasons: 

(1)  The  criteria  for  system/mission  success  and  failure  are 
defined  explicitly  in  terms  of  operational  objectives. 

(2)  Formulation  of  the  effectiveness  modal  requires  that 
system  functions  and  quantitative  system  requirements 
be  related  to  mission  profiles  and  demands  within  real¬ 
istic  operational/environmental  requirements  and 
conditions. 

(3)  The  inherent  and  extrinsic  possibilities  for  system 
failure/degradation  and  restoration  are  related  to  the 
ability  of  the  system  to  meet  operational  demands. 

(4)  Design  characteristics,  system/support-system 
configuration  and  operating/maintenance  procedures  are 
related  to  sys tem/mission  functional  performance  in  such 
a  way  that  detail  design  as  well  as  major  system  alterna¬ 
tives  can  be  evaluated  in  terms  of  quantitative  impact  on 
overall  effectiveness. 

(5)  As  a  result  of  these  attributes,  a  properly  formulated 
system  effectiveness  model  provides  an  essential  tool 
in  the  system  value  approach  for  evaluating  alternative 
approaches  to  system  design/configuration,  and  thus 
provides  the  rational  basis  (1)  for  the  iterative  definition 
of  systems  in  successively  greater  detail,  (2)  for  the 
trade-off  procedures  required  for  system  optimization 
and  (3)  for  interpreting  the  results  of  test  and  evaluation. 
Thus,  the  effectiveness  modal  provides  the  design- 
decision,  evaluation  capability,  along  with  the  cost 
model  and  considerations  of  military  worth,  required 

by  the  system  analysis  and  engineering  process. 
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The  effectiveness  model  together  with  the  associated  definition  of 
criteria  and  objectives  plays  a  central  role  in  the  system  analysis  and 
engineering  process  in  several  ways.  First,  it  provides  the  framework 
within  which  major  alternative  concepts  are  evaluated  relative  to  each 
other  and  to  operational  objectives  and  missions.  During  and  after 
selection  of  a  concept,  it  provides  the  basis  for  quantitative  develop¬ 
ment  of  system  requirements  and  their  allocation  to  subsystems  and 
lov/er  levels  within  the  system  hierarchy.  Thus,  effective  communica¬ 
tion  between  user  and  producer  is  promoted.  Next,  it  provides  the 
means  whereby  the  producer  can  evaluate  the  developing  system  in 
terms  of  requirements  to  determine  (1)  whether  the  proposed  designs/ 
configurations  are  likely  to  meet  operational  (user)  requirements,  and 
(2)  the  need  for,  and  effect  of,  design  changes.  It  provides  the  frame¬ 
work  within  which  test  results  can  be  evaluated  by  the  user  relative  to 
operational  requirements.  And  finally,  the  effectiveness  model  pro¬ 
vides  the  basis  for  assessing  system  effectiveness  during  the  deploy¬ 
ment  phase  and  thus  aids  in  the  continuing  process  of  defining  the 
requirements  for  system  improvement,  growth  and  evertual  obsoles¬ 
cence  in  terms  of  changing  threats  and  state  of  the  art. 

System  effectiveness /system  value  (SE/SV)  analysis  is  required 
whenever  the  selection  of  alternative  designs,  policies  or  plans  is 
required,  or  whenever  optimization  of  a  set  of  continuous  parameter 
values  must  be  performed,  on  the  basis  of  effectiveness  or  system 
value.  SE/SV  analysis,  at  an  appropriate  level,  is  an  integral  part  of 
each  evaluation  step  in  the  system  engineering  process  and  is  thus 
required  during  each  iteration  until  the  design/development  process  is 
completed.  SE/SV  analysis  is  also  an  integral  part  of  the  integrated 
logistic  support  process,  which  is  itself  part  of  the  system  engineering 
process  during  the  first  three  acquisition  phases;  and  is  required  to 
evaluate  the  impact  on  system  effectiveness  and  system  value  of  support 
system  alternatives.  Thus,  the  SE/SV  analysis  approach  and  the  SE/ 

SV  models  are  the  decision  tools  of  choice  for  use  throughout  the 
acquisition  life  cycle  and  provide  a  rational  basis  for  decisions  which 
must  be  made  by  system  engineering  and  ILS  managers. 

Ideally,  the  iterative  process  of  system  development  employs  a 
basic  algorithm  more  or  less  like  the  one  shown  in  Figure  3-1  . 

(1)  Requirements /objectives  at  any  level  of  system  definition  are  used 
to  guide  (2)  conceptualization  and  characterization  of  one  or  more 
approaches  to  satisfying  the  requirements/objectives.  This  provides 
a  structure  within  which  (3)  criteria/measures  and  decision  rules  ran 
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Figure  2-1.  Idealized  process  for  iterative  system  development 


be  established  by  which  it  can  be  decided  whether  or  not  the  require¬ 
ments /objectives  are  satisfied.  (4}  Measurement  and  evaluation 
(including  modeling,  prediction,  model  exercise,  etc.  )  provide  the 
basis  either  for  (5a)  a  decision  among  alternatives  if  they  are  discretely 
definable  or  (5b)  optimization  in  case  of  a  continuous  range  of  possi¬ 
bilities.  This  establishes  the  system’s  definition  at  a  given  level  after 
which  requirements  are  allocated  to  the  next  level  of  definition  and  the 
process  is  reiterated. 

In  system  engineering,  steps  (1),  (3),  (4),  (5)  and  (6)  are  coopera¬ 
tive  activities  of  the  design  and  analysis  functions,  with  analysis  taking 
the  lead,  whereas  step  (2)  is  almost  entirely  a  design  function.  In 
practice,  the  design  engineering  groups  and  the  specialty  groups  such 
as  reliability,  maintainability,  logistics,  human  factors,  etc.  ,  have 
both  design  and  analysis  functions.  The  design  functions  are  collec¬ 
tively  designated  the  "engineering51  function  within  system  engineering, 
while  the  analysis  functions  are  collectively  designated  the  "system 
analysis"  function  within  system  engineering  and  are  coordinated  among 
the  various  groups  through  use  of  the  system  effectiveness/system 
value  approach.  Thus,  the  design  function  is  responsible  for  defining 
approaches  to  meeting  effectiveness  requirements,  the  analytic  function 
evaluates  the  approaches  using  system  effectiveness/system  value 
models,  and  (ideally)  both  functions,  through  recommendations  to 
management,  participate  in  making  decisions  and  allocating  further 
requirements.  Figure  3-?.  translates  this  algorithm  into  a  system- 
effectiveness  /system- value  engineering  process. 


Figure  3-2.  System  Effectivenocs/ System  Value  Prediction/Decision/Impiementation  Cycle 


Although  the  process  is  rarely  seen  in  this  ideal  form  in  practice, 
it  is  nevertheless  true  that  each  step  must  be  performed  at  least  implic¬ 
itly  if  systems  are  to  be  developed  which  are  optimum  in  terms  of 
mission  objectives  and  effectiveness /value  criteria.  Significant  cur¬ 
tailment  of  any  of  the  critical  steps  (or  their  transposition,  such  as 
in  "after-the-fact"  analysis)  will  likely  result  in  less  than  optimal 
systems  or  even  in  inability  to  meet  operational  requirements. 


Implementation  of  the  system  effectiveness  approach  in  a  syst>  m 
program  is  aimed  at  producing  the  system  with  greatest  value,  hence 
this  chapter  considers  effectiveness  assurance  within  a  system  value 
context.  Effectiveness  is  a  measure  of  the  ability  of  the  system  to 
accomplish  the  mission  objectives.  System  value  studies  are  concerned 
with  achieving  a  combination  of  mission  utilization,  resource  use  and 
attained  effectiveness  that  is  best  according  to  a  selected  criterion  of 
value.  Resource  use  represents  the  expenditure  of  dollars,  manpower, 
material,  time,  facilities,  etc.  ,  required  for  the  development,  opera¬ 
tion  and  support  of  a  system.  Such  studies  are  carried  out  concurrently 
and  integrally  with  the  development  of  missions  and  systems  in  order 
to  select  among  a  set  of  alternatives  and  to  optimize  the  selected 
system. 

System  value  extends  the  cost-effectiveness  concept  by  consider¬ 
ing  the  military  worth  of  the  primary  and  secondary  missions  of  a 
system,  thus  taking  into  account  differences  in  missions  which  can  be 
performed  by  competing  systems.  If  all  systems  being  considered 
perform  the  same  set  of  missions  at  the  same  rate  and  over  the  same 
duration,  then  for  purposes  of  system  selection,  relative  mission  worth 
can  be  set  at  unity  across  all  systems  and  the  concept  reduces  to  the 
familiar  one  of  cost  effectiveness. 

There  are  four  major  decision-making  levels  at  which  system 
value  analysis  can  be  meaningfully  applied.  The  first  is  to  establish 
the  mission(s)  and  objectives  by  considering  overall  defense  goals, 
geopolitical  and  environmental  factors,  and  economic  and  technological 
capabilities.  This  is  generally  coordinated  at  the  DoD  level. 

The  second  is  to  synthesize  alternative  system  concepts  and  to 
select  the  preferred  system(s)  during  the  Conceptual  Phase.  This  is 
primarily  the  responsibility  of  the  procuring  agency. 


The, third  is  to  optimize  the  preferred  system  to  obtain  optimal 
use  of  resources  while  satisfying  system/mission  requirements.  This 
happens  during  the  Validation  and  Full-Scale  Development  Phases  and 
is  the  joint  responsibility  of  procuring  agencies  and  contractors. 

The  fourth  is  to  monitor  the  operational  effectiveness  of  the  sys¬ 
tem  and  to  provide  the  rationale  for  modification,  improvement  and 
eventual  replacement.  This  happens  during  the  Deployment  and  Opera¬ 
tional  Phase  and  is  primarily  the  responsibility  of  the  user. 

This  manual  primarily  addresses  the  second  and  third  decision 
levels  and  the  tasks  required  to  implement  the  system  effectiveness/ 
system  value  approach.  Typical  implementation  tasks  and  their  inter¬ 
relations  are  shown  diagrammatically  in  Figure  3-2.  The  process  is 
iterative  and  proceeds  from  gross  to  more-detailed  levels  along  with 
the  system  engineering  process  of  which  it  is  part. 

Task  1.  Analyze/Define  Mission(s) 

System  effectiveness /system  value  (SE/SV)  analysis  begins  with 
definition  and  analysis  of  the  mission(s)  in  order  to  derive  or  clarify 
objectives,  system  functions  and  quantitative  system  requirements. 

This  should  proceed  systematically  from  identification  of  mission  objec¬ 
tives  and  characterization  of  the  threat  to  epecification  of  mission  seg¬ 
ments  and  profiles,  and  result  in  mission  scenarios  with  functional 
requirements,  time  lines  and  quantitative  mission  demands  on  system 
performance  capabilities.  Ideally,  probability  distiibutions  of  mission 
demands  (performance  levels,  response  limes,  durations,  frequencies, 
etc. )  should  be  derived.  Derivation  of  a  range  of  mission  possibilities 
provides  the  basis  for  deriving  system  functions  and  quantitative  system 
requirements  along  both  performance  level  and  time  dimensions. 

The  mission  definition  task  is  critical  to  all  the  rest  of  the  tasks, 
and  is  a  prerequisite  to  specification  of  criteria  and  formulation  of  the 
effectiveness  model.  In  too  many  projects,  inadequate  definition  and 
analysis  of  the  mission(s)  results  directly  in  poorly  formulated  criteria 
and  unrealistic  effectiveness  models,  and  systems  which  are  inadequately 
responsive  to  real  operation?!  requirements.  This  is  particularly  true 
for  such  factors  as  required  response  times  which  determine  organiza¬ 
tional  configuration,  maintenance  and  system  ready  policies;  frequencies 
and  durations  of  demand  which  have  implications  for  reliability,  main¬ 
tainability,  manning  and  spares;  and  other  time -dependent  factors. 
Inadequate  consideration  is  also  given  to  secondary  or  alternative 
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missions  in  many  cases,  resulting  in  a  less -than- optimal  system 
where  a  relatively  small  additional  investment  would  have  considerably 
extended  the  range  of  mission  utilization  and  hence  increased  overall 
system  value.  Adequate  mission  definition  and  analysis  identify 
these  factors  in  advance  so  they  can  be  reflected  in  system  require¬ 
ments,  rather  than  waiting  for  their  appearance  in  the  field  and  the 
more-costty  consequences  of  modification  or  simply  living  with  the 
oroblem. 


Task  2.  Identify  Resources  and  Constraints 


In  addition  to  defining  the  technical  requirements  of  a  system,  it 
is  also  necessary  to  define  the  resources  available  for  development  and 
procurement,  which  primarily  act  as  constraints  and  thus  bound  the 
system  value  model  inputs.  The  four  principal  types  of  resources 
which  must  be  considered  are:  budget,  project  manning,  industry 
capacity  and  technology.  In  addition,  time  is  a  resource,  and  limita¬ 
tions  on  development  time  must  be  considered  along  with  the  change  in 
military  worth  resulting  from  delays.  This  is  generally  governed  by 
the  phasing  of  other  systems,  and  projections  of  threat,  developing 
enemy  capabilities,  and  growth  in  our  own  technology  which  may  result 
in  obsolescence  of  the  current  concept. 


Other  types  of  "constraints"  are  derived  from  mission  considera¬ 
tions  or  as  a  result  of  model  exercise  and  apportionment  of  goals/ 
requirements  to  lower-level  system  elements.  These  are  constraints 
on  effectiveness  variables  or  parameters  (such  as  downtime,  MTBF, 
etc. )  or  on  lower-level  cost  elements  resulting  from  analysis.  These 
are  derived  rather  than  primary  constraints  and  are  part  of  Tasks  4, 

7,  9  or  11. 


The  primary  non- resource  constraint  (requirement)  which  is 
nevertheless  primary  and  should  be  considered  at  this  point  is  the 
minimum  allowable  effectiveness  below  which  the  mission  and  system 
could  no  longer  be  justified.  This  is  a  difficult  determination  to  make 
in  most  instances  unless  a  higher-level  total  force  model  was  used  to 
generate  requirements,  in  which  case  the  apportioned  probability  of 
mission  success  or  system  value  index  is  consistent  with  that  of  other 
force  elements.  Frequently,  however,  setting  a  minimum  effectiveness 
requirement  is  a  matter  of  judgment.  In  either  case,  preliminary 
effectiveness  requirements  should  be  set  at  the  highest  organizational 
levels  as  a  result  of  system  value  consideraticns  and  threat  criticality, 
and  included  in  early  planning  documents.  If  not,  then  the  project 
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manaeer  has  no  alternative  but  to  set  what  he  feels  to  be  an  achievable 
goal  considering  the  state-of-the-art  and  gain  approval  after  the  fact. 

In  any  event  effectiveness  requirements  should  be  set  as  soon  as 
possible  and  included  in  the  RFP  for  contractor  guidance. 

Task  3.  Synthesize/Describe  System  Alternatives 

In  this  task,  objectives,  system  functions  and  quantitative  system 
requirements  are  considered  in  synthesizing  one  or  more  system  con¬ 
cepts  or  configurations.  Resource  constraints  are  used  as  bounds  for 
screening  and  narrowing-  down  the  list  of  possibilities.  The  depth  of 
system  description  depends  on  the  phase  of  tne  system  life  cycle. 

The  number  of  alternatives  identified  depends  on  the  available 
technology,  budget  and  schedule.  Practically  speaking,  alternatives 
should  usually  be  limited  to  a  few  of  the  most-likely  contenders.  In 
some  cases,  only  one  basic  system  concept  may  be  considered,  and 
alternatives  are  .onsidered  only  for  lower-level  system  elements  during 
the  optimization  process. 

The  level  of  system  description  depends  on  the  phase  of  develop¬ 
ment,  but  should  include  at  least  the  major  attributes  influencing 
effectiveness  and  cost.  Any  differences  among  alternatives  in  mission(s) 
utilization  must  also  be  included.  Physical  characteristics  such  as 
weight,  volume,  shape,  energy  levels,  mechanical  and  electrical  pack¬ 
aging  and  environmental  capabilities  should  be  included  as  well  as  per¬ 
formance  characteristics  such  as  accuracy,  speed,  range,  capacity, 
power  output,  discrimination,  etc.  Depending  on  the  phase  of  develop¬ 
ment,  some  level  of  definition  of  reliability,  maintainability,  human 
performance  assumptions  or  capability  s,  manning,  maintenance  con¬ 
cept  and  the  like  should  also  be  included.  In  general,  the  major  effec¬ 
tiveness  factors  of  capability,  availability  and  dependability  should  be 
kept  in  mind  in  order  to  generate  a  checklist  of  system  functional  attri¬ 
butes  which  will  enable  a  model  to  be  formulated  on  the  basis  of  the 
mission  and  system  description. 

The  Design  Disclosure  for  Systems  and  Equipment  (DDSE)  dis¬ 
cussed  in  Appendix  B  forms  an  ideal  basis  for  system  description  in  a 
form  which  is  readily  communicated  to  the  system  effectiveness  analyst 
as  well  as  to  other  program/project  members. 


Task  4.  Specify  Decision  Criteria  and  SE/SV  Measures 


Concurrently  with  and  immediately  following  Tasks  1  through  3, 
system,  mission  and  resource  definitions  are  analyzed  in  order  to 
specify  decision  criteria  and  SE/SV  measures.  Actually,  this  is  an 
iterative  process  since  the  nature  of  the  criteria  depends  upon  the  level 
of  consideration.  The  highest  level  criterion  is  system  value,  and  the 
decision  criterion  may  be  expressed  as  ‘’accept  the  sv3terr»  (or  lower- 
level)  alternative  which  has  the  highest  value  subject  to  the  constraint 
that  system  value  is  equal  to  or  greater  than  x  with  probability  P.  and 
that  resource,  effectiveness  and  other  constraints  are  satisfied."  In 
general,  this  is  the  preferred  form  for  stating  any  criterion. 

The  first  step  is  to  identify  the  hierarchy  of  measures  that  corres¬ 
pond  to  important  attributes  of  the  system  which  influence  value  and  its 
components  --  system  effectiveness,  costs /penalties  and  military  worth 
of  missions.  Some  of  the  major  factors  were  presented  in  Figure  2-1 
for  system  value  and  in  Figure  2-6  for  system  effectiveness.  Having 
decided  which  measures  are  important,  the  processes  of  allocation 
(apportionment)  and  prediction  are  carried  out.  These  accompany 
successive  formulation  and  exercise  of  the  SE/SV  model(s)  with  each 
iteration  at  levels  of  increasing  definition  and  provide  quantitative 
specification  of  the  required  or  apportioned  values  of  the  measures. 
Higher-level  criteria  or  values  of  measures  (requirements!  come  from 
mission  and  resource  considerations.  Lower-level  criteria  are  derived 
by  the  joint  use  of  prediction  and  apportionment.  This  proceeds  as 
follows : 

1.  Th.e  system  is  defined  at  the  next- lower  level. 

2.  Measures  influencing  value  and  effectiveness  at  that  level 
are  identified. 

3.  The  SE/SV  model(s)  are  formulated  at  that  level. 

4.  Preliminary  prediction  of  the  quantitative  values  of  the 
measures  is  carried  out,  or  goals  are  allc  .ated,  for 

the  system  as  defined. 

5.  The  model  is  exercised  to  determine  whether  predicted 
or  preliminary  allocated  values  would  be  consistent  with 
meeting  higher  level  criteria  with  acceptable  probability. 
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If  so,  these  are  established  as  the  apportioned 
requirements  at  that  level.  If  not,  feasiV”«y  studies 
are  performed  to  determine  what  measu.  . .  attributes) 
could  be  improved  to  meet  higher-level  criteria.  These 
are  then  specified  as  the  apportioned  requirements 
and  lead  to  modification  of  the  system  definition. 

6.  When  the  concurrent  process  of  design  (system  defini¬ 
tion)  and  analysis  (evaluation)  results  in  an  agreed- 
upon  system  which,  at  the  given  level  of  definition, 
meets  higher-level  requirements,  the  process  is  taken 
to  the  next-lower  level  and  repeated.  However,  deci¬ 
sions  about  criteria  and  measures  at  each  level  are 
regarded  a3  tentative  and  subject  to  reiteration  depend¬ 
ing  on  the  results  of  lower-level  analyses.  While 
lower-level  analyses  are  performed  on  an  element  by 
element  basis,  they  are  finally  used  as  input  to  the 
overall  SE/SV  model  to  determine  whether  apportioned 
criterion  values  are  consistent  with  overall  requirements. 

The  specification  of  decision  criteria  and  SE/SV  measures,  then, 
is  a  dynamic,  iterative  process  which  evolves  along  with  the  system 
definition  and  SE/SV  model(s)  formulation  and  exercise. 

Task  5.  Identify  Variables  and  Specify  Accountable  Factors 


Each  SE/SV  measure  at  a  given  level  is  a  function  of  primary  and 
support  system  variables  which  must  be  interrelated  within  the  frame¬ 
work  of  the  model(s).  Actually,  the  specification  of  measures  and 
systematic  examination  of  the  system  and  mission  will  generally  suggest 
these  variables.  Many  of  the  variables  become  SE/SV  "measures"  or 
"criteria"  at  the  next  or  lower  levels  of  definition. 

Specification  of  accountable  factors  is  a  management  task  which 
provides  for  the  assignment  of  criteria/measures/variables,  with 
stated  quantitative  desigr  goals  or  requirements,  to  organizational 
entities.  This  suggests  a  natural  basis  for  project  organisation  which 
corresponds  to  program  objectives,  system  definition  and  effectiveness 
tasks.  It  also  provides  the  management  control  and  visibility  neces¬ 
sary  for  SE/SV  assurance. 


Task  6.  Identify  Data  Sources 


Specification  of  criteria,  measures  and  variables  also  provides  a 
specification  of  the  type  of  data  required  as  model  input.  Identification 
of  sources  for  these  data  is  a  crucial  and  sometimes  difficult  task. 

Very  frequently,  alternative  sources  of  varying  quality  exist,  depending 
on  the  available  budget  and  leadtime.  The  choice  of  source  in  such 
instances  will  depend  on  data  criticality,  which  in  turn  is  only  verifiable 
through  sensitivity  analysis  involving  model  exercise.  Thus,  initial 
specification  of  data  source  may  need  to  be  revised  later. 

Typical  data  sources  include: 

•  Test  and  evaluation  data 

•  Historical  data  from  similar  systems 

•  Field  data  collection  systems 

•  Laboratory  experiments  using  system  mockups, 
simulators,  etc. 

•  Computer  simulation  models 

•  Published  "standard"  data 

•  Subjective  "expert"  judgment  data. 

Data  source  selection  will  also  depend  on  acquisition  phase.  For 
instance,  early  gross  estimates  may  be  satisfied  by  subjective  judg¬ 
ments  from  "experts"  and  historical  data  from  similar  systems,  while 
later  estimates  may,  depending  on  sensitivity,  require  laboratory  experi¬ 
ments  or  computer  simulation,  and  finally,  the  use  of  T&E  or  field  data. 

A  question  which  must  always  be  answered  is  "what  are  the  expected 
gains  considering  the  cost  of  acquiring  data  from  source  x?  "  The  major 
consideration  in  answering  the  question  is  the  reduction  in  uncertainty 
related  to  data  source  improvement.  Along  a  related  dimension,  the 
amount  of  such  data  to  be  collected  determines  the  statistical  factor 
of  risk.  The  factors  of  risk  and  uncertainty  must  always  be  evaluated 
in  specifying  data  source.  The  requirement  is  to  relate  (1)  risk  and 
uncertainty  inherent  in  data,  to  (2)  risk  and  uncertainty  of  decisions,  to 


(3)  criticality  of  decisions  to  overall  value  and  effectiveness.  In  other 
words,  the  "cost  of  being  right"  must  be  weighed  against  the  "cost  of 
be:-  ^g  wrong.  " 

Task  7.  Construct  Models 

The  tasks  up  to  this  point  provide  the  basis  for  constructing  the 
SE/SV  model(s).  The  models  to  be  constructed,  of  course,  depend  on 
what  is  to  be  done  with  them.  In  typical  large-scale  system  programs, 
these  uses  will  lead  to  construction  of  overall  system  value,  system 
effectiveness  and  cost  models.  At  lower  levels,  trade-off  models  are 
usually  formulated  for  purposes  of  suboptimization  since  manipulation 
of  the  total  model  will  generally  be  unwieldy  and  unnecessary  for  arriv¬ 
ing  at  many  decisions.  Typical  trade-offs  include: 

•  Reliability  vs.  Maintainability 

•  Vulnerability/Survivability  vs.  Performance 

•  Safety  vs.  Performance 

•  Speed  vs.  Range 

•  Rate  vs  Accuracy 

•  Etc. 

Trade-off  models  generally  allow  for  the  variation  of  two  or  more 
parameters  (such  as  MTBP  and  MTTR)  to  determine  the  values  which 
permit  some  objective  to  be  met  (such  as  availability)  at  least  cost  or 
penalty  (where  cost  or  penalty  may  be  dollars,  time,  weight,  man¬ 
power,  etc.  ). 

Either  as  a  built-in  feature  of  the  model(s)  or  as  a  separate  model, 
risk,  uncertainty  and  their  separate  and  combined  effect  on  decisions 
should  also  be  considered. 

One  of  the  decisions  will  be  whether  to  construct  special  models 
or  to  use  one  of  the  general  models  which  are  available  for  some  pur¬ 
poses.  This  will  depend  on  budget,  time  and  relevance  of  available 
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models.  Relevance  is  determined  by  considering  the  properties  of  the 
model  in  relation  to  the  system/mission  in  terms  of  the  following 
qualities: 

•  Assumptions  (reasonable?  ) 

•  Adequacy  (includes  all  major  variables?  ) 

•  Risk  and  Uncertainty  (adequate  treatment  of  probabilistic 
aspects?  ) 

•  Validity  (has  model  been  validated  through  actual  use, 
i.  e.  ,  does  it  yield  field -verifiable  outputs?) 

Questions  which  should  be  answered  about  a  model  include: 

•  Consistency  (are  results  consistent  when  major  param¬ 
eters  are  varied,  especially  to  extremes?  ) 

•  Sensitivity  (do  input- variable  changes  result  in  output 
changes  that  are  consistent  with  expectations?  ) 

•  Plausibility  (are  results  plausible  for  special  cases 
where  prior  information  exists?  ) 

•  Criticality  (do  minor  changes  in  assumptions  result  in 
major  changes  in  the  results’) 

•  Workability  (does  the  model  require  inputs  or  computa¬ 
tional  capabilities  that  are  not  available  or  within  the 
bounds  of  current  technology?  ) 

•  Suitability  (is  the  model  consistent  with  the  objectives, 
i.  e.  ,  will  it  answer  the  right  questions?  ) 

Modern  approaches  to  SE/SV  and  related  modeling  for  military 
systems  depend  heavily  on  stochastic  or  probabilistic  rather  than 
deterministic  formulations  since  they  are  to  be  used  for  making  deci¬ 
sions  under  risk  ?,nd  uncertainty.  Older  deterministic,  models  are 
Therefore  rarely  adequate.  Some  of  the  available  models  and  modeling 
approaches  are  discussed  in  the  Appendices.  The  general  character¬ 
istics  of  effectiveness  models  were  diocussed  in  Section  2.  2. 


Task  8.  Acquire  Data 


Early  identification  of  variables  and  data  sources  should  be  made 
to  allow  sufficient  time  for  data  acquisition  by  the  specialty  groups 
responsible  (reliability,  maintainability,  logistics,  human  factors, 
cost,  performance  engineering  groups,  etc.)-  Acquisition  of  data  in 
computer  input  format  will  frequently  expedite  Task  9  (Estimate  Model 
Parameters).  Data  should  generally  be  collected  in  raw  or  semi- 
reduced  form  where  multiple  uses  exist.  This  is  particularly  true  for 
numerical  data  descriptive  of  stochastic  processes  characterized  by 
probability  distributions.  In  such  eases,  either  raw  data  or,  at  mini¬ 
mum,  the  first  four  distribution  moments  (or  distribution  function)  and 
sample  size  plus  qualifications,  data  collection  method,  conditions  or 
assumptions,  should  be  reported.  In  the  case  of  subjective  "expert" 
judgments,  estimates  should  be  in  the  form  of  "high,  "  "low"  (or  upper 
and  lower  percentiles)  and  "most  likely"  (or  mean/mode /median) 
estimates,  accompanied  by  qualifications  of  estimators,  accompanying 
comments,  their  feelings  of  confidence,  and  methods /questionnaires / 
instructions  accompanying  the  collection  of  the  estimates.  This  is 
important  for  proper  interpretation  of  the  results  and  for  arriving  at 
estimates  of  "subjective  probability  distributions"  and  degree  of 
uncertainty. 

Data  acquisition  is  followed  by  cr  concurrent  with  the  next  task. 
Task  9.  Estimate  Model  Parameters 


Along  with  or  following  data  acquisition,  the  values  of  model 
parameters  to  be  input  in  Task  11  (Exercise  Models)  are  estimated. 
Ideally,  these  should  be  in  the  form  of  probability  distributions  where 
the  inherent  process  contains  variability.  Confidence  intervals  or 
sampling  distributions  should  be  associated  with  point  estimates  of 
means  derived  from  samples,  and  "fiduciary  limits"  or  "subjective 
confidence"  associated  with  subjective  estimates.  These  are  discussed 
in  Section  2.3.2. 

Task  10.  Specify  Schedule 


Specifying  (he  schedule  is  a  management  task  generally  carried 
ovit  with  the  aid  of  a  PERT  network  or  similar  device.  The  main  objec¬ 
tive  is  to  schedule  design  and  analysis  tasks  so  they  are  complementary 
and  such  that  analytical  results  will  have  maximum  impact  on  design. 
This  almost  always  means  getting  the  analytical/modeling  tasks  initiated 
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much  earlier  in  the  development  cycle  than  has  generally  been  the  case 
in  past  programs.  It  is  also  necessary  to  schedule  prerequisite  ana¬ 
lytical  tasks  such  as  Task  4,  5  and  6  with  enough  leadtime  to  provide 
timely  inputs  to  the  model  formulation  and  exercise  tasks. 

Task  11.  Exercise  Models 


In  this  task,  the  models  are  exercised  to  provide  the  inputs 
required  by  the  decision  process.  If  the  models  are  well-formulated 
and  data  acquisition  is  successful,  the  models  can  be  exercised  in  highly 
flexible  ways  in  response  to  all  the  important  questions  requiring  ana¬ 
lytic  resolution.  In  addition  to  overall  estimates  of  system  value, 
effectiveness  and  cost,  numerous  trade-off  analyses  and  optimization 
studies  will  be  available  at  the  beck  and  call  of  project  management 
and  engineers.  An  important  output  for  decision  purposes  will  be  the 
risks  and  uncertainties  associated  with  each  output  together  with  impor¬ 
tant  assumptions. 

Prior  to  or  concurrent  with  producing  such  outputs,  the  various 
parameters  and  variables  should  be  varied  over  their  ranges  to  deter¬ 
mine  sensitivity  of  the  overall  measures  to  variation  in  input  values. 
This  is  an  important  indication  of  criticality  of  the  associated  system 
attributes,  and  is  roughly  a  measure  of  the  justifiable  investment  e'ther 
in  design  activity,  in  quantification  effort  to  reduce  uncertainty,  or 
in  "over-design"  to  reduce  risk. 

Two  principal  uses  of  models  are  for  evaluation  and  prediction. 
Evaluation  provides: 

•  Surveillance  of  current  system  status  against  quanti¬ 
tative  system  requirements 

•  Feedback  upon  the  efficacy  of  the  management  decision 
and  program  control  process 

•  A  means  of  determining  system  weaknesses  or  potential 
problem  areas 

9  A  point  estimate  of  system  value  and  effectiveness  which 
includes  all  pertinent  factors  within  a  uniform  framework. 
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Prediction  provides  decision  aids  through: 


•  Comparative  prediction/evaluation  of  competing  system 
configurations  and  problem  solutions 

•  Calculations  of  the  effects  of  risk  and  uncertainty  expressed 
as  confidence  levels,  parameter  variation  studies  and 
changing  requirements  analysis. 

The  use  of  a  model  involves  the  following  steps: 

Perform  model  checks 
Calculate  values  of  criterion  measures 
Do  trsde-o££a  within  constraints 
Compare  calculations  with  standard  ot  reference 
Calculate  parameter  sensitivity  curves 
Calculate  risk 

Calculate  effect  of  uncertainty 

•  Interpret  runs. 

Task  12.  Prepare  Management  Reports 


The  results  of  model  exercises  (output  values  and  risks/uncer¬ 
tainties)  should  be  presented  in  summary  form  together  with  recommen¬ 
dations  and  important  qualifications.  The  summary  reports  should  be 
backed-up  by  more-detailed  reports  (separate  or  as  appendices)  which 
cover  methods,  model  structure,  assumptions,  parameter  values  and 
summarized  input  data,  etc.  Where  outputs  have  associated  probability 
distributions,  they  should  be  plotted  graphically  so  that  management 
can  readily  choose  a  level  of  risk  themselves  rather  than  having  the 
analyst's  choice  of  risk  implicit  in  the  output.  Accompanying  these 
reports  with  disclosure  in  the  DDSE  format  (see  Appendix  B)  will 
considerably  expedite  interpretation. 
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Task  13.  Make  Decision 


Although  the  analyst  may  make  recommendations,  the  ultimate 
decision  must  rest  with  project  management  in  order  to  reflect  extra- 
analytic  factors,  which  may  in  some  instances  be  significant  to  eventual 
user-acceptance.  However,  a  well- formulated  and  executed  analysis 
will  provide  an  ordered  and  rational  basis  for  decision  which  should  con¬ 
siderably  enhance  decision-making  power  and  lend  credibility  which 
would  otherwise  be  missing  or  difficult  to  provide. 

Task  14.  Implement  Decision 

Implementing  decisions  re  suiting  from  SE/SV  analyses  is  similar 
to  implementing  decisions  from  design  review  or  any  other  significant 
program  event.  Of  course,  without  this  step  the  point  of  the  SE/SY 
analysis  is  lost.  Nevertheless,  this  is  frequently  the  weak  point  in 
system  programs  and  results  in  much  useless  effort  not  to  speak  of  a 
"bad  taste"  where  analysis  is  concerned.  Although  the  analyst  can  do 
much  to  encourage  implementation  through  well-documented,  well- 
formulated,  relevant  results,  implementation  depends  ultimately  upon 
the  project  manager  and  his  understanding  of  the  significance  of  the 
analyses  and  resulting  recommendations /decisions. 

Task  15.  Change  Analysis 

The  implementation  of  a  decision  based  on  SE/SV  considerations 
implies  a  change  in  one  or  more  of  the  following: 

•  Schedule 

•  Model(s) 

•  System  definition 

•  Requirements  (measures,  criteria,  etc.). 

Each  iteration  should  be  accompanied  by  Change  Analysis  against  each 
of  these  areas.  The  result  will  be  a  monitoring  of  the  net  effect  of 
each  decision  and  the  accomplishment  of  program  surveillance. 


CHAPTER  FOUR:  SYSTEM  EFFECTIVENESS  IN  THE  RDT&E 

PROGRAM  AND  ACQUISITION  CYCLE 


The  processes  required  to  develop  Navy  systems  are  consistent 
and  well-defined  in  basic  terms  although  terminology,  documentation 
and  division  into  phases  may  vary  from  time  to  time.  At  any  given 
time,  the  Navy  RDT&E  program  is  a  structured  procedure  in  which  a 
continuous  dialogue  is  maintained  between  the  user  (CNO/CMC)  and  the 
producer  (CNM/contractors)  from  definition  of  broad  objectives  to  sys¬ 
tem  conceptualization,  definition,  development,  production,  deployment 
and  operation.  This  process  is  summarized  in  Figures  4-1  and  4-2. 

Figure  4-1  shows  the  normal  documentation  flow  that  takes  place 
in  the  RDT&E  planning  process.  The  dialogue  is  continuous  between 
the  user  and  producer.  It  moves  gradually  and  orderly  from  long-range 
planning  on  through  the  point  of  introducing  new  hardware  and  weapon 
systems  to  the  men  with  our  operating  forces. 

Figure  4-2  shows  the  major  phases  of  the  acquisition  cycle  in 
relation  to  RDT&E  activities.  Note  the  utilization  of  the  DCP  (Develop¬ 
ment  Concept  Paper)  and  DSARC  (Defense  Systems  Acquisition  Review 
Council)  combination  as  a  key  for  going  into  systems  development  and 
procurement.  System  Value  with  its  components  of  System  Effective¬ 
ness,  Cost  and  Military  Worth  provide  the  key  to  these  decisions  from 
the  earliest  phases  of  the  acquisition  cycle.  A  key  document  related 
to  this  process  is  the  DCP,  which  is  a  memorandum,  issued  by  OSD, 
normally  prepared  by  DDR&E,  coordinated  with  .he  Services,  express¬ 
ing  decision*  c/n  initiation  of,  or  changes  to,  major  R&D  programs. 

A  DCP  is  a  relatively  concise  document  (20  pages  or  less)  which  makes 
explicit  assumptions  concerning  (1)  agreed-upon  problem  or  threat, 

(2)  development  time  frame,  (3)  priority,  (4)  force  levels  contemplated 
and/or  measures  of  merit,  or  effectiveness,  which  will  be  used  to 
evaluate  and  compare  alternative  or  competitive  systems.*  Thus,  the 
"effectiveness  approach"  is  initiated  at  the  highest  levels  of  the  planning 
processes  and  at  its  onset.  Continual  development  of  the  effectiveness 


Navy  Research,  Development,  Test  &  Evaluation  Program,  Head¬ 
quarters,  Kaval  Material  Command,  March  1972.  See  also  Depart 
ment  of  the  Navy  RDT&E  Management  Guide,  NAVSO  P-2457, 

1  July  1972. 


-67- 


■  -  u USU.W&  -fwor 


£  R0T8.E  FROG  RAM:  PLANN!NGtU$*R>I  _ 

'• '  '  •'-  --  -• 

USER  CNO/CMC 


STRATEGIC  1  OPERATIONAL  MATERIAL 

TACTICAL  STUDIES  CA^ABUIIY  PERFORMANCE 

“  ,  P£S  OBJECTIVES  OBJECTIVES 

IROAO  NAVAL  OIJECIIVES  /  S4  /  / 

LONG  RANGE  PUNS  (LRP|  /  lRp 


ADO 


POTENTIAL 


PRODUCER 

l  /  CONCEPTS  & 

l  FORECAST/OF  TECHNOLOGY 
i  NTPIt)  NTP  11,111 

t  /  AmM  Rk.  TmJes 


B*sic  Rtsearch 


1.1 


L 


Expferatwy  D«v. 

CmcEft 

6.2  fat gtWSftottiwt  1.3 


DECISION  POINT 
(OSARC) 


APPROVAL 


Stwfcts 


M  . 


CNR- 


CNM 


Figure  4-1.  RDT& E  Program:  Planning,  User/Producer 
Interaction 


.  RDT&E PROGRAM 


OSD  MHESTONES 

PHASES:  CONCEPTUAL 


ACQUISITION  CYCLE 
2 

VALIDATION  {  FUU  SCALE 
|  DEV. 

•  UPDAJe  DCP  I  •  UPDATE  DCP 


MORE—* 

MONITOR  PROGRESS  ft  COSTS 

*APP10V£  MAJOR  CHMGtS 


•  Otv.  COSCIPT  PAH*  I 

™an  *  SYSTEMS  ACQUISITION  REVIEW  COUNCIL  (DSAP.C}  DECISIONS 


Tigure  4-2.  Acquisition  cycle 


-68- 


model  and  us  exercise  for  decision /evaluation  throughout  all  phases  a.  e 
essential  to  successful  execution  of  the  RDT&E  program. 


The  user-producer  relation  is  more  analogous  to  a  relation 
between  cooperating  independent  business  organizations  than  to  tradi¬ 
tional  military  relations.  Plans  are  the  result  of  negotiation. between 
the  two  interests.  Through  this  process  trade-offs  are  made  that  will 
result  in  the  maximum  military  capability  for  the  operating  forces 
within  the  limits  of  the  resources  available. 

The  principal  documents  used  in  the  user-producer  dialogue  are 
shown  in  Figure  4-1.  The  process  involves  a  continuous  interaction 
between  operational  requirements  and  their  spokesmen,  and  technical 
and  scientific  possibilities  and  their  spokesmen.  It  is  a  continuing, 
iterative  interchange. 

The  Chief  of  Naval  Operations  is  responsible  for  the  preparation 
of  a  General  Operational  Requirement  (GOR)  for  each  functional  war¬ 
fare  and  support  area.  GORs  usually  result  from  rather  extensive  long- 
range  strategic  and  tactical  studies.  These  documents  state,  in  rela¬ 
tively  broad  but  significant  terms,  the  capabilities  the  Navy  needs 
within  each  area.  For  guidance  in  making  trade-offs  in  weapons  design, 
the  GOR  should  indicate  the  relative  importance  of  the  needed  capabili¬ 
ties.  In  the  past,  performance  capabilities  have  been  adequately  stated 
in  the  GORs;  however,  other  considerations  that  constitute  system 
effectiveness  --  reliability,  maintainability,  etc.  --  have  not  always 
been  given  adequate  attention.  System  effectiveness  guidance  must  be  pro¬ 
vided  for  the  entire  system  at  the  GOR  stage,  for  here,  along  with  the 
evolving  DCP,  is  where  the  thinking  and  planning  for  total  system  effec¬ 
tiveness  begin. 

In  some  cases  the  using  agency  issues  a  document  concerning  a 
narrower  requirement,  a  Tentative  Specific  Operational  Requirement 
(TSOR).  This  document  states  the  need  for  achieving  a  particular 
operational  capability  and  outlines  the  identifiable  system  characteris¬ 
tics  necessary  to  fulfill  the  requirement.  The  TSOR  defines  the  desired 
performance  goals  and  provides  additional  information  needel  to  weigh 
alierr .itives  and  mike  the  tradeoffs  required  for  an  optimum  system. 

The  producer  response  to  either  the  GOR  or  TSOR  is  a  PTA  (Proposed 
Technical  Approach).  PTAs  are  developed  by  the  Naval  Material  Com¬ 
mand  to  propose  technically  feasible  alternative  methods  of  accom¬ 
plishing  objectives  set  forth  in  a  GOR  or  TSOR.  The  PTA  should  be 


fully  responsive  to  the  GOR  or  TSOR;  therefore,  the  quality  of  the  PTA 
depends  directly  on  the  quality  of  the  GOR  or  TSOR.  In  addition  to  other 
mandatory  requirements  of  the  PTA,  the  governing  OPNAV  and  DoD 
directives  require  that  the  PTA  analyze  and  compare  the  operational 
effectiveness  of  the  proposed  alternate  development  approaches  in  terms 
of  performance,  reliability,  operability,  and  maintainability,  and  clearly 
indicate  the  basis  of  the  comparison,  such  as  previous  experiments, 
extrapolation,  or  conjecture. 

The  user  reviews  what  is  presented  in  the  PTA  and  decides  on  one 
of  the  following  alternatives: 

1.  Study  the  requirement  further 

2.  Begin  feasibility  studies,  further  experimentation,  or 
both 

3.  Begin  an  engineering  or  operational  development  effort 

4.  Terminate  development  effort  in  the  specific  area 

If  alternative  1  is  chosen,  the  process  returns  to  the  strategic 
and  tactical  study  phase  and  usually  results  in  revisions  to  the  GOR  or 
TSOR.  If  alternative  2  is  chosen,  the  user  interests  develop  and  pro¬ 
mulgate  an  Advanced  Development  Objective  (ADO).  If  alternative  3 
is  chosen,  the  user  interests  develop  and  promulgate  a  Specific  Opera¬ 
tional  Requirement  (SOR).  In  the  case  of  alternative  4,  all  effort  pro¬ 
posed  in  the  PTA  is  terminated,  which  usually  results  in  the  action 
indicated  for  alternative  1,  although  on  occasion  the  requirement  will 
remain  unmodified  and  essentially  dormant  until  research  effort  devel¬ 
ops  new  technical  approaches  to  be  incorporated  in  a  superseding  PTA. 

If  alternative  2  (ADO)  or  alternative  3  (SOR)  is  chosen,  the  pro¬ 
ducer  prepares  a  Technical  Development  Plan  (TDP).  However,  there 
is  a  distinct  difference  between  a  TDP  that  responds  to  an  ADO  and  one 
that  responds  to  an  SOR.  In  the  case  of  an  ,*vDO,  the  effort  defined  by 
the  TDP  is  either  directed  toward  demonstration  of  feasibility  of 
approach(es)  or  experimentation  at  the  breadboard  level.  This  effort, 
if  successful,  leads  to  an  SOR  and  a  responding  TDP. 

The  TDP  responding  to  an  SOR  represents  the  essential  comple¬ 
tion  of  the  Conceptual  Phase.  The  most  important  end  produci  of  the 
Conceptual  Phase,  u  comprises  the  plan  for  fulfilling  the  operational 


-70- 


requirements  of  the  user.  The  goal  of  a  TDP  is  a  balanced  and  inte¬ 
grated  effort  for  optimizing  operational  effectiveness,  total  cost,  and 
early  availability. 

With  development  of  the  TDP,  the  necessary  RDT&E  planning  for 
subsequent  phases  of  the  system  is  established;  if  planning  has  been 
adequate,  only  a  minimum  of  TDP  updating  will  be  required  later. 

4.2  System  Effectiveness  and  the  System  Acquisition  Process 

The  role  of  system  effectiveness  analysis  during  the  acquisition 
process  is  to  enable  the  program  manager  to  restructure  the  allocated 
goals  to  the  system  level,  thereby  allowing  system  decisions  to  be  made 
by  higher  management.  The  allocation  process  adds  a  new  dimension 
to  management.  Dynamic  life  cycle  management  and  Integrated  Logistics 
Support  then  become  practical  goals.  The  vehicle  fo**  the  structuring 
process  is  the  system  effectiveness  model,  frequently  disossed  but 
too  often  not  used  until  after  the  fact. 

The  following  sections  describe  the  role  of  system  effectiveness 
analysis  in  the  system  acquisition  process.  It  will  be  seen  that  system 
effectiveness  techniques  are  intended  to  aid  the  project  manager  in 
decision-making  by  presenting  him  with  organized  information,  and  to 
assist  him  in  assigning  task  priorities  by  highlighting  critical  areas 
within  his  project.  The  discipline  of  System  Effectiveness  is  not  a 
replacement  for  managerial  judgment;  rather,  it  supplies  a  basis  for 
better  and  more  timely  decision*. 

4.  2.  1  Conceptual  Phase 

The  Conceptual  Phase  includes  the  activities  preceding  a  decision 
to  carry  out  Validation  (system  definition/aclvanced  development)  and  is 
conducted  at  the  discretion  of  the  Navy  without  specific  approval  by  OSD. 
During  this  phase,  the  technical,  military  and  economic  bases  for  an 
acquisition  program  are  established  through  comprehensive  systems 
studies  and  experimental  hardware  development  and  evaluation.  The 
Conceptual  Phase  is  highly  iterative.  Its  stages  overlap  rather  than 


occurring  sequentially;  however,  flowing  from  interacting  inputs  of 
operational  needs  and  technology,  generally  the  following  3tages  occur: 

•  Identification  and  definition  of  conceptual  systems 

•  Analysis  (threat,  mission,  effectiveness,  feasibility, 
risk,  cost,  trade-offs,  etc.) 

•  Experimentation  and  test  (of  operational  requirements, 
key  components,  critical  subsystems  and  marginal 
technology) 

The  outputs  of  *he  Conceptual  Phase  are  alternative  systems 
(including  a  preferred  system)  and  their  associated  program  charac¬ 
teristics  (costs,  schedules,  and  operational  parameters)  based  on  a 
combination  of  analyses,  experiments  and  test  results. 

The  initiation  of  successful  syslem  development  programs  in  the 
U.S.  Navy  is  becoming  increasingly  difficult  in  view  of  the  rapid  tech¬ 
nological  changes  to  be  coped  with  and  the  growth  of  required  program 
documentation.  Success  depends  upon  many  complex  factors  such  as 
the  following: 

•  Determination  of  threat  profiles  and  their  translation 
into  system  requirements  and  constraints 

•  Status  and  understanding  of  performance  parameters, 
resource  estimates,  and  error  budgets  of  exploratory/ 
advanced  development  projects. 

9  Understanding  of  the  activities  reqr.ired  to  satisfy  the 
directives,  requirements,  and  instructions  of  the  DoD/ 

Navy  management  system 

•  Availability  of  people,  facilities,  techniques,  and  data 
to  support  required  activities. 

The  integration  of  the  above  factors,  and  others,  for  the  specific 
purpose  of  initiating  an  engineering  development  program  takes  place 
during  the  Conceptual  Phase.  Much  work  has  to  be  done  to  prcv'de  effi¬ 
cient  conceptual  capability.  ...  cohesive  marriage  of  the  design  and 
analysis  techniques  is  required.  The  results  of  conceptual  studies  will 
h.  '•e  a  m:  or  impact  upon  the  cost  and  responsiveness  of  future  Naval 
systems 


4.  2.  1 .  1  Candidate  System  Definition 

In  an  ideal  situation  the  Conceptual  Phase  v/ould  progress  front 
the  recognition  of  a  threat,  to  a  number  of  approaches,  to  candidate 
concepts,  and  then  to  candidate  systems.  This  idealized  order  is  sel¬ 
dom  realized.  The  ’ack  of  orderliness  in  the  real-world  evolution  of 
the  process  can  pose  extreme  problems  if  not  approached  in  a  disci¬ 
plined  manner. 

Basic  to  a  disciplined  approach  is  the  recognition  that  these  stages 
of  progress  are  simply  differing  degrees  of  precision  in  defining  the 
system.  In  other  words,  the  descriptive  parameters  become  pro  ..  .  j- 
sively  better  defined  for  each  step  in  the  evolution  from  approach  t 
candidate  system. 

During  the  earlier  stages,  system  functions  are  quite  broadly 
defined.  The  gross  functions  are  progressively  structured  as  groups  of 
subfunctions,  each  with  its  associated  inputs  and  outputs.  Structuring 
continues  until  the  candidate  system  has  oe  jn  defined.  This  approach 
permits  comparative  evaluation  of  competing  candidate  systems, 
regardless  of  their  relative  stage  of  evolution. 


4.2.  1.2  Preferred  System  Selection 

The  p-eferred  system  selection  process  must  take  into  account 
considerations  other  than  system  effectiveness.  /\u.ong  these  are  sys¬ 
tem  value  and  risk.  The  cost  analysis  component  of  system  value  is 
based  on  two  cojt  estimates  associated  with  each  function  in  the  system 
effectiveness  model.  One  estimate  covers  the  cost  of  acquisition,  the 
other  covers  the  cost  of  utilization  or  ownership.  The  former  includes 
the  RDT&E  costs,  prorated  over  the  anticipated  production  quantitv,  as 
well  as  the  production  ard  installation  costs  per  system.  The  latter 
includes  all  operating,  maintenance,  and  support  costs  of  the  system. 
These  costs  can  be  used  in  connection  with  trade-off  analyses,  or  they 
can  be  aggregated  and  associated  with  the  s/stem  value  index  and  used 
as  partial  determinants  in  preferred  system  selection.  Other  partial 
determinants  useful  in  preferred  system  selection  are  comparisons  of 
system  effectiveness  indices  with  manpower  ard  lead-time  requirements. 

Along  with  the  formulation  and  exercise  of  the  system  value  model 
and  its  submodels  for  effectiveness,  cost,  development  time  and  the 
like,  a  formal  analysis  of  risk  and  uncertainty  should  be  initiated.  This 
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requir -s  that  sources  of  risk  and  uncertainty  associated  with  each  inpt  t 
be  identified  and  that  estimates  of  the  range,  variability  and/or  confi¬ 
dence  of  inputs  be  made.  These  are  then  combined  appropriately  during 
the  analysis  to  provide  the  combined  risk  and  uncertainty  due  to  all 
sources  for  each  of  the  major  parameters  --  effectiveness,  cost,  devel¬ 
opment  lead-time,  etc.  This  provides  a  critical  input  to  the  system 
selection  decision  process,  and  may  lead  to  an  entirely  different  deci¬ 
sion  than  would  have  been  made  without  it.  In  addition  to  identification 
of  sources  of  risk  and  uncertainty,  a  most  useful  form  of  output  of  an 
analysis  of  risk  and  uncertainty  is  a  probability  distribution  (objective 
or  subjective)  of  each  measure  or  parameter.  The  project  manager  is 
thereby  given  a  "quantitative"  definition  of  risk  and  uncertainty  in  terms 
of  the  probability  (under  '■he  stated  assumptions)  that  time,  cost  or 
effectiveness  will  meet,  exceed  or  fall  short  of  requirements  or  goals  -- 
and  by  how  much.  It  is  also  a  direct  measure  of  the  combined  effects 
of  the  quality  and/or  variability  of  input  data  on  any  decisions  made  on 
the  basis  of  exercising  the  models. 

The  results  of  the  effort  to  this  point  are  formally  organized  as 
a  PTA  and  submitted  through  appropriate  channels  to  the  Chief  of  Naval 
Operations. 

The  system  value  methodology  is  based  on  the  evaluation  of  sys¬ 
tem  effectiveness,  military  worth,  cost  and  their  degradation  as  a 
function  of  the  time  to  acquire  the  system  and  of  operational  life-span. 
Neither  military  worth  nor  degradation  is  directly  measurable.  How¬ 
ever,  they  can  be  assigned  numeric  judgment  valuations  by  military 
experts.  The  military  worth  index  is  an  evaluation  of  the  mission  to 
be  accomplished  by  the  system.  If  all  candidate  systems  accomplish 
identical  missions,  the  military  worth  valuation  can  be  set  at  unity,  and 
only  the  effects  of  ccjt,  time-to-acquire  and  operational  life-span  will 
be  considered  in  the  evaluation.  On  the  other  hand,  if  one  or  more  of 
the  candidate  systems  are  capable  of  accomplishing  additional  mission? 
or  different  combinations  of  missions,  the  indices  of  military  worth 
should  reflect  the  differences  as  military  judgment  may  deem  appro¬ 
priate.  The  net  effect  of  the  system  value  methodology  is  to  provide  a 
military  judgment  coefficient  to  assist  in  system  selection. 

The  actual  selection  is  suggested  by  the  candidate  system  with  the 
highest  index  of  system  value.  However,  this  suggestion  is  not  absolute 
Modeling  assists  in  the  decision-making  process  and  is  not  a  substitute 
for  managerial  judgment.  Indeed,  judgment  may  result  in  a  decision  at 
this  stage  that  more  than  one  preferred  system  will  enter  Validation. 


Formal  Conceptual  Phase  activity  generally  terminates  with 
proposal  of  a  "system  option"  in  Part  HI  of  the  Navy  Technological 
Projections  (NTP)  or  with  the  response  to  a  Tentative  Specific  Opera¬ 
tional  Requirement  (TSOR)  and  the  issuance  of  an  Advanced  Develop¬ 
ment  Objective  (ADO).  In  practical  application,  then,  the  Conceptual 
Phase  includes  the  early  conception  of  new  systems  (which  help  provide 
focus  for  Exploratory  Development  planning)  and  the  program  execution 
required  to  provide  the  technology  necessary  to  make  the  concept  tech¬ 
nically  feasible. 

The  preferred  system(s)  having  been  selected,  one  step  remains 
prior  to  Validation.  To  the  project  manager,  this  is  one  of  the  most 
critical  steps,  and  his  first  major  test  as  a  manager.  He  must  demon¬ 
strate  that  he  has  met  all  of  the  prerequisites  to  obtain  DSARC  review 
and  SECDEF  approval  to  enter  into  Validation. 

If  not  approached  in  a  well-organized  manner,  this  demonstration 
can  be  a  time-consuming  and  frustrating  exercise.  However,  the  Sys¬ 
tem  Effectiveness /System  Value  methodology,  with  the  associated 
models,  provides  the  ordered  approach  and  the  demonstration  vehicle. 
Using  these  models,  the  manager  can  define  the  system(s)  in  terms  of 
technical  goals  and  criteria,  trade -off  evaluations,  and  priorities  of 
effort,  together  with  the  associated  confidence  levels. 

Application  of  the  Navy  system  value  methodology  throughout  the 
Conceptual  Phase  of  the  system's  life  cycle  places  the  manager  in  an 
unambiguous  position.  If  he  can  define  his  system  sufficiently  well  to 
exercise  the  models,  it  is  probable  that  his  system  is  soundly  con¬ 
ceived  and  that  the  model  completion  in  itself  will  demonstrate  his  meet¬ 
ing  of  the  prerequisites  for  Validation.  On  the  other  hand,  inability  to 
provide  minimal  input  requirements  for  model  analysis  and/or  to  pro¬ 
vide  clear  system  definition  is  a  strong  indication  that  prerequisites 
have  not  been  met.  Having  successfully  demonstrated  accomplishment 
of  prerequisites  during  the  Conceptual  Phase,  the  manager  uses  the 
essential  inputs  to  prepare  the  Request  for  Proposal  (RFP)  needed  to 
cover  the  contracted  effort. 

4.  2.  2  Validation  Phase 

This  is  the  phase  in  which  the  major  program  characteristics 
(technical,  cost  and  schedule),  through  extensive  analysis  and  hardware 
development,  are  validated  and  is  often  identified  with  Advanced  Devel¬ 
opment.  It  is  preferred  to  rely  on  hardware  development  and  evaluation 
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rather  than  paper  studies  alone,  since  this  provides  a  better  definition 
of  program  characteristics,  higher  confidence  that  risks  have  been 
revived  or  minimized  and  greater  confidence  in  the  ultimate  outcome. 
Nevertheless,  effectiveness  analysis  plays  a  critical  role  since  it  pro¬ 
vides  a  structured  vehicle  for  interrelating  and  evaluating  the  informa¬ 
tion  developed  as  a  result  of  hardware  development,  and  organizes 
information  in  a  form  suitable  for  updating  the  DCP,  and  for  DSARC 
review  leading  to  the  decision  to  enter  Full  Scale  Development.  In  an 
idealized  case,  this  pnase  ends  when  a  "brass  board"  model  has  been 
demonstrated  successfully. 

The  Validation  Phase  is  a  period  of  major  concern  to  the  project 
manager,  although  the  burden  of  proving  performance  and  responsive¬ 
ness  rests  on  the  contractor  (private'industry  or  Government  labora¬ 
tory)  who  has  been  selected  as  a  qualified  participant  essentially  on  the 
basis  of  proposals. 

In  many  respects  the  application  of  the  system  value  methodology 
to  Validation  parallels  its  application  in  the  Conceptual  Phase.  There 
are,  however,  some  significant  differences.  In  time  sequence,  the 
application  of  the  system  value  concept  during  the  Conceptual  Phase,  as 
discussed  in  Section  4.  2.  1,  is  applicable  if  the  term  "Contractor's 
Proposed  System"  is  used  in  lieu  of  "Candidate  System.  " 

The  candidate  and  preferred  system(s)  having  been  defined  in  the 
Conceptual  Phase,  a  sensitivity  analysis  is  performed  with  the  effec¬ 
tiveness  model.  This  analysis  will  indicate  the  limiting  parame.ers  and 
priorities  for  each  element  of  the  system  model,  which  are  expressed 
in  terms  of  technical  goals  or  requirements.  The  range  of  the  permis¬ 
sible  parameters ,  properly  related  to  estimates  of  state-of-the-art 
capabilities,  establishes  the  degree  of  criticality  of  the  element. 

Along  with  the  sensitivity  analysis,  the  analysis  of  risk  and  uncer¬ 
tainty  performed  during  the  Conceptual  Phase  should  be  updated,  extended 
and  carried  out  to  greater  levels  of  detail. 

The  system{s)  definition  and  the  critical  system  effectiveness 
parameters  are  incorporated  into  a  Rrque3t  for  Proposal,  which  is 
transmitted  to  the  contractor(s)  as  a  guide  for  proposed  approaches  to 
Validation.  The  system(s)  definition  provides  for  the  initiation  of  Vali¬ 
dation,  the  equivalent  of  Candidate  System  Definition  in  the  Conceptual 
Phase.  In  addition  to  guidance  for  the  contractor(s),  the  definition  of 
critical  system  effectiveness  parameters  provides  the  criteria  for  eval¬ 
uating  contractor  proposals. 
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As  with  other  aspects  of  the  system  effectiveness  discipline,  the 
definition  of  critical  system  effectiveness  parameters  is  not  static.  The 
process  of  refinement  started  in  the  Conceptual  Phase  continues  in 
Validation.  As  a  result  of  the  analysis  of  contractor  or  laboratory  pro¬ 
posals,  the  system(s)  definition  is  refined,  and  the  critical  parameters 
are  better  defined  by  the  inputs  received  from  the  responses  to  the  RFP. 
This  sharpening  becomes  most  important  to  the  project  manager  during 
latter  stages  of  Validation  and  during  Full-Scale  Development 

The  project  manager  may  exercise  little  control  over  the  Valida¬ 
tion  effort.  However,  progress  reports  under  the  contracts  do  provide 
definition  and  validating  data.  As  these  data  are  received,  reiterative 
exercise  of  the  system  effectiveness  model  provides  a  significant  mea¬ 
sure  of  the  progress  being  realized. 

While  the  critical  system  effectiveness  parameters  can  be  defined 
initially  during  the  errly  Conceptual  Phases,  they  reach  much  greater 
definition  in  the  latter  stages  of  Validation  during  the  analysis  effort. 
They  provide  the  essential  framework  for  the  decision  to  enter  into  Full- 
Scale  Development. 

The  definition  of  these  parameters  at  this  point  in  the  evolutionary 
cycle  of  a  system  must  be  sharpened  to  the  point  where  the  project 
manager  can  demonstrate  the  following: 

«  The  operational  goals  and  technical  goals  are  in 
agreement 

•  The  technical,  and  hence  operational,  criteria  can 
be  met 

•  The  financial  and  schedule  factors  are  credible 

•  The  development  risks  are  acceptable 

•  A  definitive  Full-Scale  Development  contract  can 
be  entered  into  with  the  best-qualified  contractor. 

To  demonstrate  the  foregoing,  not  only  must  the  parameters  be 
clearly  and  concisely  defined,  but  they  must  be  quantitatively  inter¬ 
related.  This  requires  highly  structured  system  models  in  terms  of 
functional  block  diagrams  with  associated  characteristics  values  and  a 
completely  structured  System  Effectiveness/System  Value  model  with 


which  to  analyze  and  evaluate  the  system  models.  The  former  is  an 
output  of  Validation  contractor  efforts.  The  latter,  however,  is  largely 
the  result  of  the  efforts  of  the  project  manager's  staff  or  research/ 
analysis  contractor.  The  success  or  failure  of  Validation  will  be  deter¬ 
mined  by  the  degree  of  completeness  of  the  model  and  the  degree  to 
which  its  structuring  conforms  to  the  real-world  situation. 

If  the  System  Effectiveness /System  Value  model  does  approximate 
leality  successfully,  the  parameters  can  be  interrelated,  and  the  exer¬ 
cise  of  the  model  for  each  of  the  competing  systems  produced  by  the 
contractor(s)  provides  a  framework  for  source  selection  and  demon¬ 
strates  the  validity  of  entering  into  Full-Scale  Development,  continuing 
with  further  definition  or  advance3  development  effort,  or  abandoning 
the  project. 

In  addition  to  its  use  as  a  decision-making  tool,  the  model  also 
serves  another  function  during  this  period.  The  sharply  defined  critical 
effectiveness  parameters  provide  the  checklist  for  completing  the  speci¬ 
fication  for  Full-Scale  Development.  This  is  particularly  important  in 
that  one  of  the  principal  objectives  of  the  Validation  process  is  to  assure 
that  a  complete  and  unambiguous  specification  is  developed  for  the  Full- 
Scale  Development  effort. 

4.2.3  Full-Scale  Development 


Through  the  process  of  Validation,  the  project  manager  has  been 
establishing  a  frame  of  reference  to  define  the  system,  its  technical 
goals  and  criteria,  and  the  measures  by  which  its  effectiveness  in  terms 
of  its  mission  life  costs  can  be  evaluated.  Having  established  this  frame 
of  reference,  he  must  now  address  himself  to  obtaining  assurance  of 
achieving  an  effective  system. 

During  Full-Scale  Development,  the  weapon  system  including  all 
of  the  items  necessary  for  its  support  (training  equipment,  maintenance 
equipment,  handbooks  for  operation  and  maintenance,  etc.)  is  designed, 
fabricated  and  tested.  The  intended  output  is  a  "hardware  model"  whose 
effectiveness  has  been  proven  experimentally  together  with  the  documen¬ 
tation  needed  for  inventory  use.  An  essential  activiiy  of  the  develop¬ 
ment  phase  is  test  and  evaluation,  both  that  conducted  by  contractors 
and  that  conducted  by  the  Service.  Documentation  of  the  Full-Scale 
Development  Phase,  including  the  results  of  effectiveness  analysis,  pro¬ 
vides  tne  basis  for  updating  the  DCP  and  convening  DSARC  leading  to 
initiation  of  Production/Deployment. 
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The  ultimate  evaluation  of  the  Full-Scale  Development  Phase 
occurs  during  the  test  and  evaluation  of  the  developed  system.  If  the 
system  model  and  system  effectiveness  analytic  model  are  valid  and 
adequately  defined,  the  system  should  meet  its  test  and  evaluation 
successfully,  and  the  project  manager  will  have  been  successful. 

If  the  system  is  not  satisfactory,  the  models  have  yet  another 
function.  The  data  accumulated  during  T&E  should  be  inserted  into  the 
models.  The  models  should  then  be  exercised  and  the  results  analyzed 
to  identify  problem  areas.  These  should  then  be  recorded  and  made 
available  to  other  project  managers  to  assist  them  in  avoiding  similar 
errors.  At  the  same  time,  a  closed-loop  management  system  should 
be  implemented  to  correct  the  problems. 

If  the  project  is  to  be  continued,  whether  or  not  the  T&E  is  suc¬ 
cessful,  the  T&E  data  are  inserted  in  the  models  to  sharpen  further 
the  definition  of  technical  goals  and  criteria  and  to  validate  the  data  for 
the  production  baseline  and  production  specification.  Here,  again,  the 
models  serve  to  guide  the  effort  and  to  assure  the  project  manager  that 
the  baseline  (specification)  is  complete  and  defined  as  sharply  as  the 
aggregate  experience  will  permit.  This  is  a  necessary  exercise, 
whether  or  not  the  R&D  contractor  is  also  the  initial  production 
contractor. 

4.2.4  Production  Phase 

When  the  system  has  passed  the  test  and  evaluation  and  has  been 
approved  for  service  use,  the  project  manager  must  produce  the  sys¬ 
tem  and  introduce  it  into  the  operational  forces.  In  the  past,  this 
transition  from  Research  and  Development  to  Production  has  meant 
turning  the  project  over  to  a  new  team-  all  too  frequently  involving  a 
great  deal  of  learning  for  the  new  team,  time  losses,  and  a  loss  of 
experience  and  data. 

Two  factors  could  provide  safeguards  against  these  traditional 
difficulties.  The  first,  the  project-manager  concept,  includes  pro¬ 
visions  for  keeping  the  management  team  intact.  The  same  manage¬ 
ment  team  that  was  responsible  for  R&D  should  have  some  continuing 
responsibility  for  production.  Thus  the  time  loss  involved  in  learning 
the  system  is  eliminated.  The  second,  use  of  both  simulation  and  ana¬ 
lytic  models,  provides  a  methodology  for  experience  and  data  retention 
The  formal  structuring  and  recording  of  data  provide  a  high  degree  of 
assurance  that  both  experience  and  data  will  be  retained. 
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When  viewed  objectively,  the  demands  placed  on  the  project 
manager  for  changes  in  configuration,  cost,  and  schedule  differ  little 
in  concept  from  the  trade-off  analyses  performed  during  the  Conceptual 
or  Validation  Phases.  Indeed,  the  same  tools,  the  system  model  and 
the  analytic  models  for  effectiveness  and  value,  can  be  used.  Actually, 
since  the  model  values  have  now  been  more  sharply  defined  through  the 
introduction  of  experimental  data  during  Validation  and  Full-t-'cale 
Development,  the  validity  of  the  models  as  decision-making  aids  should 
be  very  high. 

4.  2.  5  Deployment  Phase 

An  important  function  of  sy3tem  effectiveness  during  Fleet  opera¬ 
tions  is  the  retrieval  of  data  for  future  programs.  One  significant 
attempt  to  provide  a  portion  of  these  data  from  operating  units  is  the 
MDCS  (Maintenance  Data  Collection  System)  carried  out  under  the  Navy 
3M  Program,  Attempts  should  be  made  to  structure  the  MDCS  data 
formats  in  such  a  way  that  the  requisite  inputs  to  the  system  effective¬ 
ness  model  can  be  obtained  from  the  MDCS  without  additional  reporting 
requirements.  The  project  manager  would  then  have  available  to  him 
the  main  body  of  field  data,  which  could  then  be  introduced  into  the 
models. 


Other  sources  of  data  are  also  available,  such  as  the  Failure  Rate 
Data  (FARADA)  Program  which  provides  failure  rate,  failure  mode, 
and  test  background  data  on  both  electrical  and  mechanical  parts  and 
components;  and  the  Government-Industry  Data  Exchange  Program 
(GIDEP),  which  was  established  in  I960  to  minimize  duplicate  testing 
through  the  interchange  of  environmental  test  data  and  related  technical 
information. 


Field  data  are  needed  for  two  principal  reasons: 

•  They  provide  the  real-life  validating  information  on  the 
project  manager's  past  decisions.  Through  this  valida¬ 
tion  effort  he  can  determine  the  adequacy  of  weighting 
and  other  judgment  factors  that  were  applied  during  the 
preceding  phases.  An  added  return  ” 5  the  recording  and 
sharing  of  these  evaluations  with  prc  ect  managers  for 
other  systems  under  development  or  or  superseding 
systems.  In  this  application  of  the  s\  .  ..  n  effectiveness 
methodology,  the  Navy  can  receive  substantial  benefits 
in  experience  retention. 


•  These  data  can  also  be  used  to  establish  a  decision 
baseline  for  determining  the  need  for  so-called  pro¬ 
duct  improvements  in  operating  systems,  and  for 
evaluating  proposed  changes.  Costly  changes  and 
changes  of  questionable  return  may  result  from  use 
of  inadequate  or  incomplete  data. 

In  the  operational  phase,  as  in  the  preceding  phases,  the  discipline 
of  the  system  effectiveness  approach  guards  against  making  decisions  on 
the  basis  of  inadequate,  incomplete,  or  unrelated  data  --  principally 
through  the  visibility  that  modeling  techniques  give  to  the  ramifications 
of  the  variations  in  data  inputs.  While  the  system  effectiveness  approach 
is  by  no  means  a  panacea  and  certainly  not  a  substitute  for  sound  judg¬ 
ment,  it  does  provide  a  structured  discipline  that  substantially  increases 
the  probability  that  the  project  manager  will  have  as  inputs  to  delibera¬ 
tions  the  factors  necessary  to  assure  that  he  makes  the  right  decisions. 
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APPENDIX  A 


AN  ILLUSTRATIVE  EFFECTIVENESS  MODEL  AND  COMPUTATION 


Suppose  a  mission  demand  which  may  occur  at  any  random  point 
in  time  requires  that  a  system  be  available  within  0.  25  hours  to  per¬ 
form  two  tasks,  A  and  B,  in  sequence  within  a  combined  accuracy,  9, 
of  0.  90  or  better;  that  no  more  than  one  failure  may  occur  after  mission 
start  with  resultant  downtime,  t^,  not  exceeding  0.  5  hours;  that  the 
total  task  time  (excluding  restore  time  during  the  mission)  may  not 
exceed  tj^  hours;  and  that  failure  to  meet  any  of  these  criteria  results 
in  failure  of  the  system  to  meet  mission  objectives.  The  various  alter¬ 
natives  are  diagrammed  in  Figure  A-l  in  a  form  which  simplifies  the 
task  of  formulating  a  system  effectiveness  model.  Failure  and  success 
paths  are  shown  with  system  states  and  tasks  in  relation  to  them.  To 
formulate  the  model,  expressions  or  estimates  must  be  derived  for 

(1)  conditional  probabilities  at  the  defined  branching  points;  (2)  proba¬ 
bility  distributions  ox  restore  and  task  times;  (3)  the  probability  distri¬ 
butions  of  task  accuracy;  and  (4)  the  mission  time,  t ivl  (or  its  distri¬ 
bution  if  it  is  variable). 

At  this  point,  it  should  be  noted  that  constructing  a  diagram  like 
Figure  A-l  requires  mission/threat  analysis  in  order  to  define  the 
criteria.  These  criteria  (effectiveness  criteria)  define  the  conditions 
for  mission  success  and  failure,  which  in  this  case  are  summarized  as 
follows: 

(1)  The  system  must  be  available  within  0.  25  hours  after 
warning 

(2)  Only  one  failure  is  allowable  during  the  mission 

(3)  If  a  failure  occurs,  downtime  during  the  mission  must 
not  exceed  0.5  hours 

(4)  Tasks  A  and  B  must  be  performed  with  an  accuracy 
of  0.  90  or  better 

(5)  Total  task  time  must  not  exceed  tj^  hours  (in  this 
illustration,  tj^  is  taken  to  be  the  mission  time). 
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mm 


Figure  A- 1 .  Illustrative  system  slate  diagram  (ESN  -  Event  Sequence  Network) 


In  addition  to  these  criteria,  mission/threat  analysis  provides  the  basis 
for  two  other  vital  bits  of  information: 

(6)  Mission  demands  occur  randomly 

(7)  Mission  time  (duration  of  demand  or  time  allowed  to 
achieve  mission  objectives)  is  tj^.  For  this  illustra¬ 
tion  it  is- assumed  that  tj^  is  constant. 

System  effectiveness  in  this  case  is  simply  the  probability  of 
mission  success.  Accounting  for  all  the  ways  the  mission  can  succeed, 
then,  is  an  essential  first  step  in  formulating  the  SE  model  with  the 
help  of  Figure  A-l.  The  first  step  is  to  convert  the  system  state  dia¬ 
gram  (also  called  an  ESN,  or  Event  Sequence  Network)  into  a  branching 
tree  diagram  such  as  the  one  given  in  Figure  A-2.  In  Figure  A-2,  it 
can  be  seen  that  two  major  branches  occur  after  Sq:  the  branch  leading 
from  Sj2  (state  12)  on  the  left  has  7  failure  paths  and  2  success  paths; 
the  branch  leading  from  Su  on  the  right  has  6  failure  paths  and  2  suc¬ 
cess  paths  --  a  total  of  13  failure  and  4  success  paths.  The  next  model' 
ing  task,  then,  is  to  write  the  overall  expression  for  Sp,  which  is  the 
probability  of  being  on  any  of  the  4  success  paths.  In  order  to  do  this, 
the  modeler  uses  two  properties  of  probability  trees: 

(1)  Given  that  the  initial  state  (Sq)  occurs,  the  probability 
of  ending  up  at  any  given  end  state  (success  or  failure) 
is  obtained  by  multiplying  all  the  intervening  state- 
probabilities  along  the  path  from  Sq  to  the  end  state. 

(2)  The  sum  of  the  probabilities  of  all  end  states  equals 
1  (unity);  and  the  total  probability  of  success  is  the 
sum  of  the  probabilities  of  all  success  end  states. 

Using  these  properties,  system  effectiveness  in  this  case  is  simply 


Success  =P12P41P21P51P61 


p  +  p  p  p 

12  41  23  51  61 
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Figure  A- 2.  Tree  diagram  of  Figure  A-l  showing  all 
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Collecting  terms,  this  becomes 


SE  -  (P12P4!  *  Pll>lP51P61,PUPM  +  P23)!'  Equation  A-l 

It  is  not  necessary,  for  present  purposes,  to  define  the  failure  states 
since 

Pr.  Failure  =1  -  Pr.  Success. 


The  next  task  is  to  estimate  or  derive  expressions  for  the  state 
probabilities  Pj  j «  Pi2»  ^21*  ^23»  ^32*  P4l»  P51  andP^j.  This  may 
be  accomplished  by  referring  back  to  .Figure  A-l,  which  states  the 
necessary  criteria,  and  proceeding  as  follows: 

PH  (Pr.  of  System  UP  Initially) 

Since  mission  demands  occur  randomly,  this  is  the  probability  of 
the  system  being  up  at  any  random  point  in  time,  which  was  defined 
earlier  as  Availability.  If  for  simplicity  we  assume  technicians,  opera¬ 
tors,  spares,  etc.  ,  are  available  immediately,  then 

P  -a  MTBF  - 

11“  i  "  MTBF  V  MTTR  ' 


Suppose  reliability  prediction  indicates  that  times  between  failures  are 
distributed  exponentially  (constant  failure  rate)  with  a  mean  time 
between  failures,  MTBF,  of  100  hours;  and  maintainability  prediction 
indicates  that  repair  time  is  distributed  lognormally  with  a  mean  time 
to  repair,  MTTR,  of  0.50  hours.  Then 


P12  (Pr.  of  System  DOWN  Initially) 


The  conditional  probabilities  at  any  juncture  add  to  I,  i.e.  ,  the 
system  is  either  up  or  down  initially.  Since  Pn  +  Pj2  =  1*  then 


P12  =  1  “  pn  =  1  -  0.9950  =  0.0050. 


P21  (Pr.  of  One  System  Failure  During  Mission) 

Assume  an  exponential  failure  density,  i.  e. ,  constant  failure 
rate,  X,  where  X  =  1 /MTBF.  The  number,  i,  of  failures  occurring  in 
a  fixed  time  interval  of  length  tj^  is  computed  from  the  Poisson  distri¬ 
bution  as  follows  (see  any  good  reliability  text): 


n,  "UM 

piV=  -S — 


Since  i  =  1 ,  then 


P 


21 


“  Pl(tM*  "  UMe 


It  was  assumed  earlier  that  MTBF  =  100  hours,  therefore 


1  —  * 

MTBF 


— — -  =  0.  01  failures  oer  hour. 
100 


Suppose  t_  =  8  hours.  Then 
M 


P21  =  0.01(8)  e'°,01(8)  =  0.0739. 
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P23  °f  No  System  Failures) 

The  probability  of  no  system  failures  is  found  by  substituting  i  =  0 
in  the  equation  for  given  previously: 


(XfcM)  e 


>tM 


-Xt. 


23 


0! 


=  e 


M 


This  is  the  familiar  equation  for  reliability  (assuming  a  series  system 

with  exponential  failures),  which  is  simply  the  probability  of  zero 

failures  during  the  mission  of  duration  t.  .. 

M 

Since  1  =  0.  01  failures  per  hour  and  t  =  8  hours 


p  _  '0.01(8) 

23  ' 


0. 9231. 


P32  (Pr.  That  Downtime  is  0.  5  Hours  or  Less) 

Jf  it  is  assumed  for  simplicity  that  the  only  source  of  downtime  is 
system  restoration  following  a  failure,  that  failures  are  detected  imme¬ 
diately,  that  repair  tasks  are  uninterrupted,  and  that  necessary  resources 
(technicians,  spares,  etc.  )  are  available  immediately,  then  downtime  is 
simply  active  repair  time.  In  this  case  we  shall  assume  that  the  Gen¬ 
eralized  Maintainability  Method  (GMM  --  see  Appendix  B)  was  used  to 
predict  the  probability  distribution  of  repair  time  so  that  the  distribution 
of  repair  time,  F(tj^),  is  given  in  graphical  form  as  shown  in  Figure  A-3. 
According  to  the  curve  for  F(tj^),  a  t^  of  0.  5  hours  or  less  has  a  proba¬ 
bility  of  0.67.  Therefore 


P.,  =  0.  67. 

id 
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P4L  (Pr*  that  Initial  Restore  iime  is  0.25  Hours  or  Less) 

If  the  system  is  down  when  warning  occurs,  it  could  be  at  any 
point  in  the  repair  process.  At  worst  the  repair  may  have  just  begun, 
and  consulting  Figure  A-3,  we  see  that  the  probability  of  repair  within 
0.  25  hours  is  0.  37.  Obviously,  if  the  repair  is  just  being  finished, 
the  probability  that  tr^  £  0.  25  hours  is  1.0.  Suppose  the  repair  is  half 
complete  --  what  is  the  probability  that  1/2  t^  £  0.25  hours?  This  is 
simply  the  probability  that  t^  ^  0.  5  hours,  which  is  0.  67.  It  is  apparent 
that  the  repair  process  is  equally  likely  to  be  at  any  point  from  beginning 
to  end  (given  that  thr  system  was  down  initially).  Therefore,  if  we 
integrate  over  all  remaining  fractions  of  task  to  be  completed  from  1  to 
0,  the  average  probability  is  an  estimate  of  P41.  An  approximation  can 
be  achieved  by  computing  maximum  allowable  repair  time  for  equally- 
spaced  increments  of  fraction-of- repair- remaining,  taking  the  asso¬ 
ciated  probabilities  from  Figure  A-3,  and  finding  their  average.  This 
computation  is  shown  in  the  following  table: 

Max.  Total  Repair  Time 
Consistent  with  Remaining 

Fraction  of  Fraction  Being  Completed  Probability  from 

Repair  Remaining  Within  0.  25  hrs.  Figure  A-3 


(f) 

(t^  =  0.25/f) 

(Pr(tR<  v^)) 

i.  0 

0.25 

0.  37 

0.9 

0.28 

0.43 

0.  8 

0.31 

0.47 

0.7 

0.  36 

0.  54 

0.  6 

0.42 

0.  60 

0.5 

0.50 

0.  67 

0.4 

0.63 

0.76 

0.  3 

0.8  3 

0.  85 

0.2 

1.25 

0.  93 

0.  1 

Z.  50 

0.  98 

0 

1 . 00 

Ave.  Pr.  0.  69 


Accordingly,  the  probability  of  repair  within  0.  25  hours  given  the  sys 
tem  was  down  at  warning  and  could  be  at  any  point  in  its  repair  cycle. 


P51  (Pr.  that  Task  Accuracy  is  0.  90  or  Greater) 


Assuming  that  we  have  experimental  or  field  data  from  which 
probability  distributions  of  task  accuracy  can  be  estimated,  the  main 
modeling  task  is  to  determine  how  these  combine  across  the  two  tasks 
to  provide  net  accuracy  of  the  system  performance  output.  The  way- 
accuracies  combine  depends  on  the  exact  nature  of  the  tasks,  their 
interaction  and  other  factors.  For  illustrative  purposes,  suppose  the 
output  of  Task  A  is  the  input  to  Task  B  and  that  accuracies  are  multi¬ 
plicative,  e.g. ,  if  Task  A  is  90  percent  accurate  and  Task  B  is  80  per¬ 
cent  accurate,  the  output  is  0.  90  x  0.  80  =  0.72  or  72  percent  accurate. 
(Certain  types  of  information  processing  provide  examples  of  this  type 
of  task. )  In  that  case,  the  overall  task  accuracy,  8,  could  very  well  be 
related  to  the  individual  task  accuracies  8^  and  9g  according  to  the 
relation 


9  =  88 
A  B 


where  8's  are  expressed  in  decimal  form.  The  problem  is  to  determine 
the  distribution  H(8)  when  the  distributions  H(8^)  and  K(8g)  are  given. 

One  common  approach  is  to  use  Monte  Carlo  methods  using  high 
speed  digital  computers.  A  pair  of  values,  8^  and  8g,  are  randomly- 
sampled  from  the  distributions  H(9^)  H(8g)  and  substituted  in  the  above 
equation  to  find  a  value  of  8.  Many  repetitions  of  this  procedure  enable 
the  distribution  H(9)  to  be  constructed. 

An  analytic  approach  can  also  be  used  which  depends  on  the  com¬ 
binatorial  properties  of  moments  and  cumularits,  which  are  readily 
derived  from  the  distributions.  The  method  has  been  called  the  Method 
of  Moments,  and  is  described  in  the  documentation  for  the  Generalized 
Maintainability  Method  (see  Appendix  B). 

Suppose  accuracies  for  Task  A  and  Task  B  are  both  distributed 
normally  with  the  following  parameters: 


Mean 

Standard  Deviation 

Task  A 

0.96 

0.01 

Task  B 

0.95 

0.01 
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It  is  assumed  that  the  distributions  represent  stochastically  indepen¬ 
dent  random  variables.  The  distributions  for  Tasks  A  and  B  are  shown 
in  Figure  A-4.  The  distribution  of  output  accuracy  is  obtained  by  com¬ 
bining  the  two  distributions  as  indicated  by  the  relation  9  =  0^9-g  given 
earlier,  and  is  also  shown  in  Figure  A-4. 

According  to  the  distribution,  the  probability  that  output  (com¬ 
bined)  accuracy  is  0.  90  or  less  is  0.  25.  Since  we  want  the  probability 
that  accuracy  is  0.90  or  greater,  we  take  1  -  0.25  =  0.75.  Therefore 


P_,  =0.  75. 

D  1 


This  is  one  of  many  possible  examples  of  how  a  performance  measure 
is  introduced  into  a  system  effectiveness  model  in  relation  to  a  mission 
criterion. 

Pftj  (Px .  that  Total  Task  Time  is  tM  or  Less) 

In  addition  to  task  accuracy,  task  time  is  another  performance 
attribute  of  the  system  which  provides  an  effectiveness  criterion.  Fre¬ 
quently,  task  time  and  task  accuracy  are  interdependent  in  some  way 
(especially  in  man-machine  systems)  and  need  to  be  considered  jointly. 
To  facilitate  the  example,  however,  we  shall  assume  this  is  not  the 
case  and  that  task  time  can  be  analyzed  independently. 

The  criterion  given  earlier  states  that  the  total  task  time  must  be 
nours  or  less.  Thus,  for  any  given  mission  performance 


t.  + 
A 


lB  -  *• 


(Pr(l*  V  'P61' 


Analysis  of  the  EST:'  in  Figure  A-l  shows  that  after  starting  the  mission, 
only  one  possible  event  path  exists  which  is  consistent  with  mission  suc¬ 


cess:  E3- 


-E4.  In  order  to  determine  the  overall  probability  that 


total  task  time  is  0.  5  hours  or  less,  it  is  necessary  to  combine  the 
individual  ta3k  (event)  -  time  distributions  to  obtain  the  overall 


99- 


4  ^  mauwk' 


distribution.  Since  the  task  times  are  additive,  the  equivalent  operation 
on  distributions  is  convolution,  denoted  by  the  asterisk  (#).  Thus 


if  t  =  +  (t,  t  and  t  are  random  variables) 

A  B  A  B 

then  G(t)  =  G(t  J  *  G(t_J. 

A  JD 


The  convolution  operation  simply  means  that  all  possible  combinations 
of  values  from  the  individual  distributions,  appropriately  weighted  by 
their  probabilities,  are  combined  into  a  new  distribution.  This  is  fre¬ 
quently  carried  out  by  simulation  procedures  using  Monte  Carlo  methods. 
The  Method  of  Moments  technique  is  more  convenient  and  less  expen  ¬ 
sive,  however,  and  was  used  to  combine  distributions  in  this  example. 
The  procedure  is  documented  as  part  of  the  Generalized  Maintainability 
Method.  ■' 

Assume  that  Tasks  A  and  B  have  lognormal  task- time  distributions 
as  shown  in  Figure  A- 5.  Their  convolution  provides  the  distribution 
shown  at  the  right  of  Figure  A- 5,  which  gives  the  probability  that  task 
time  is  any  value  _t^or  less.  Since  tj^  =  8  hours  and  the  corresponding 
probability  is  0.  96,  then 


0.  96. 


Computation  of  System  Effectiveness 


The  necessary  probabilities  have  now  been  derived  and  are  sum¬ 
marized  as  follows: 


Pu  =  0.  9950 
P  2  =  0.  0050 


Naval  Electronics  Laboratory  Center,  Generalized  Maintainability 
Method  (GMM),  Technical  Document  152,  23  November,  1971. 
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P2l  =  0.  0739 
P23  =  0.  9231 

P32  =  °*  67 

P41  ■  °-  69 
P ,,  =  0,  75 

P61  =  °'  96 


From  Equation  A-t,  system  effectiveness  is  expressed  as 

SE  =  (P  nP  ,  +  P,  )  [P  P  (p  p  +  p  \i 
12  41  ll'1  51^61^21  32  +  P23'> 


Substitution  of  the  above  values  provides 

SE  =  (0.  0050  x  0.  69  +  0.  9950)  [0.  75  x  0.  96  (0.  0739  x  0.  67  ,  0.  9231)| 
=  0.  9985  [0.  7003]  =  0.  6992. 


tn  hAn  effectiveneS3  of  °*  6992  means  that  the  system  can  be  expected 
achieve  its  mission  objectives  less  than  70  percent  of  the  time  In 

TtheTerm;  be  considered  unacceptable.  Inspection 

of  P  /  Hi!  f  1"odelushows  that  the  problem  is  the  low  value 

w51  w  3’  W  13  the  Probability  that  task  accuracy  is  0.  90  or 
,*  ®r*  ^  thl,S  1S  a  fai)‘ma"hine  system,  we  might  discover  that  the 
man  operator  is  the  weak  link  and  assign  the  problem  to  the  Human 
Factors  Department.  Various  approaches  might  be  considered,  such 
of  the °mftl0n  *°  ehminate  much  or  all  of  the  operator's  role,  redesign 
hLllv  k  11  ?d6Slgn  °{  equipment  to  make  the  task  easier,  use  of  more- 
coal  I,  i  f  operators,  etc.  Along  with  assignment  of  the  problem,  a 
i  '  n.  .  allocated.  Assuming  all  terms  except  Pt)  remain  con¬ 
stant,  and  assuming  tno  vSE  requirement  were  0.  90,  then 


0.  90  =  0.  9985  [0.  9337  P  ] 
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and 


P  =  0.  9654  (design  goal). 

Thus,  the  Human  Factors  Department  has  the  problem  of  increasing 
P5I  from  0.75  to  0.97.  If  this  cannot  be  completely  achieved,  then 
improvement  in  the  other  terms  must  also  be  considered.  In  any  event, 
trade-offs  involving  costs,  schedule  and  other  factors  should  be  carried 
out  to  decide  on  the  set  of  design  improvement  approaches  which  pro¬ 
vides  greatest  system  value. 

Referring  back  to  Figure  A-l  and  Equation  A-l  for  SE,  it  car 
readily  be  seen  that  the  probabilities  can  be  grouped  in  accordance  with 
the  partitions  Availability,  Dependability  and  Capability  as  follows: 

A  =  .Pr.  (System  Available)  =  Pjj  4  Pi2^41  =  (System 
Available  at  Warning)  +  (System  Unavailable  at 
Warning)  x  (System  Repaired  Within  Warning  Time) 

=  0.  9985 

D  =  Pr.  (System  Dependable)  =  P23  +  ^21^32  =  (No 

System  Failure)  +  (One  System  Failure)  x  (System 
Restored  Within  Allowable  Downtime)  =  0.  9726 

C  =  Pr.  (System  Capable)  =  P51P61  =■  (Task  Accuracy 
Sufficient)  x  (Task  Time  Within  Allowable  Time) 

=  0.  7200 

and 

SE  =  (A)(D)(C)  =  (0.  9985)(0.  9726)(0.  7200)  =  0.  6992. 

This  is  the  same  result  we  obtained  before.  It  is  interesting  to  note 
that  availability  of  the  system  in  this  case  is  not  simply  Ai(=Pj|)  which, 
alone,  would  have  underestimated  system  availability;  but  also  includes 
a  term  for  availability  within  warning  time  due  to  repair  of  a  down 
system.  This  illustrates  the  need  for  caution  in  using  simple  formula¬ 
tions  such  as  the  one  for  A^  without  first  constructing  the  overall  effec¬ 
tiveness  model. 
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It  should  also  be  pointed  out  that  the  simplicity  of  the  illustration 
in  itself  could  be  misleading,  since  many  practical  cases  do  not  provide 
independent  probabilities  for  A,  D  and  C  which  are  multiplicative  in  the 
simple  fashion  provided  by  the  foregoing  example.  Generally,  matrix 
calculations  or  their  equivalent  are  required  (see  the  WSEIAC  documen¬ 
tation,  for  example).  In  those  cases,  the  general  approach  used  in  the 
example  still  holds,  except  that  more-detailed  diagrams  and  models  are 
required  to  account  for  the  dependencies  among  variables.  For  this 
reason,  partitioning  of  the  model  into  A,  D  and  C  terms  is  not  generally 
recommended  at  the  overall  level,  and  the  entire  diagram  must  fre¬ 
quently  be  carried  along  to  lower  levels  of  indenture.  Unfortunately, 
adequate  discussion  of  the  required  techniques  is  beyond  the  scope  of 
this  manual.  To  avoid  pitfalls,  the  manager  should  make  sure  that 
competent  analysts  are  used  who  are  skilled  in  the  application  of  proba¬ 
bility  and  statistics  to  practical  system  modeling. 


APPENDIX  B 


THREE  EFFECTIVENESS-RELATED  PROCEDURALIZED  METHODS 

B.  1  Generalized  Maintainability  Method  (GMM) 

The  Generalized  Maintainability  Method  (GMM)  is  a  generalized 
procedure  for  the  analysis  of  system  maintainability  and  the  estimation 
or  prediction  of  time-oriented  measures  of  maintainability.  Outputs 
are  in  a  form  required  for  effec  iveness  modeling  of  the  type  discussed 
in  Appendix  A.  In  order  to  achieve  generality  and  encompass  a  maxi¬ 
mum  range  of  application  the  procedure  is  for  the  design  of  a  useful 
maintainability  model  rather  than  for  the  routine  application  of  a  pre¬ 
designed  model.  Tne  procedure  is  applicable  to  all  types  of  systems, 
to  all  acquisition  phases,  to  all  levels  of  analysis,  and  to  all  situations. 
It  may  be  used  on  electronic,  electromechanical,  mechanical,  and 
hydraulic  systems,  from  the  equipment  level  to  system  level,  and  for 
shipboard,  airborne,  or  other  operational  environments.  The  basic 
parameters  of  measurement  are  any  time-oriented  maintainability 
parameters,  corrective  or  preventive,  ranging  in  complexity  from 
active  repair  time  and  associated  man-hours  to  operational  downtime. 
Since  distributions  are  produced  if  desired,  any  parameter  or  per¬ 
centile  of  the  distribution  may  be  an  output. 

The  GMM  modeling  technique  incorporates  an  ESN  diagram 
(similar  to  Figure  A-l)  which  displays  the  alternative  sequences  of 
events  which  can  potentially  take  place  in  transitioning  from  initial 
system  states  in  which  maintenance  requirements  are  implied,  to  sub¬ 
sequent  system  states  in  which  maintenance  has  been  performed.  The 
ESN  can  be  used  to  represent  any  level  of  system,  subsystem,  or  equip¬ 
ment  complexity.  Structuring  the  various  system  assumptions  selected 
for  analysis  will  result  first  in  the  design  of  an  initial  or  overall  ESN 
(or  series  of  such  ESN's)  which  represents  the  maintenance /support 
concept  for  the  system.  Continued  application  at  lower  levels  of  inden¬ 
ture  will  result  in  detailed  ESN's  which  represent  specific  maintenance 
actions. 

The  alternative  assumptions  about  the  states  of  the  system  and 
the  possible  sequences  of  events  which  can  change  the  system  from  one 
state  to  another  produce  a  network  containing  junctures  from  which 
alternative  branches  are  defined.  The  basis  for  estimating  probabilities 
for  eacli  of  these  branches  will  depend  upon  the  type  of  branch  being 
considered.  Where  branches  depend  upon  the  existence  of  particular 


failures,  probabilities  are  obtained  from  reliability  data.  Branches 
may  depend  upon  alternative  diagnostic  strategies,  alternative  test 
facilities,  support  provisions,  and  mission  or  environmental  frequency- 
of-occurrence  data. 

The  basis  for  estimating  the  range  of  possible  durations  for  the 
maintenance  actions  or  events  will  depend  upon  the  level  of  analysis 
and  the  availability  of  retrievable,  idstorical  data.  Although  objective 
data  are  preferable,  it  is  recognized  that  the  best  available  data  will 
sometimes  be  in  the  form  of  estimates  made  by  expert,  knowledgeable 
individuals  who  are  familiar  with  the  system  in  question,  and/or  with 
the  component  maintenance  events  which  are  similar  in  other,  existing 
systems.  GMM  accepts  data  from  any  source  and  in  any  combination. 

The  initial  ESN  displays  both  the  time-consuming  maintenance 
actions  or  events,  and  the  intermediate  states  of  the  system  and  mission 
which  aid  the  analyst  in  formulating  maintainability  measures.  In  cal¬ 
culating  the  overall  time  distribution  for  these  measures,  the  initial 
ESN  is  converted  into  a  tree  diagram  (similar  to  Figure  A- 2)  which  dis¬ 
plays  the  event  sequences  in  a  form  which  is  convenient  for  tracing 
alternative  paths.  The  tree  diagram  displays  a  finite  random  process 
which  is  defined  by  the  following  characteristics: 

•  From  a  fixed  starting  point  (state),  any  one  of  a  finite 
number  of  events  can  occur. 

•  For  each  possible  event,  a  finite  number  of  subsequent 
events  can  occur. 

•  The  process  eventually  ends  in  one  of  a  finite  number 
of  well-defined  terminal  states. 

A  path  through  a  tree  diagram  or  ESN  is  a  single- thread  sequence  of 
events  leading  from  an  initial  state  to  a  terminal  state. 

Time  distributions  may  be  computed  either  by  simulation  or  ana¬ 
lytic  procedures.  The  strategy  for  calculation  of  the  combined  time 
distribution  by  analytic  procedures  is  described  in  the  GMM  documen¬ 
tation. 


Further  information  about  GMM  may  be  obtained  from  the  Naval 
Electronics  Laboratory  Center,  Code  4100,  San  Diego,  California.  Tech¬ 
nical  Document  152  (November  1911}  describing  and  illustrating  the  pro¬ 
cedure  may  be  obtained  from  the  above  source  by  qualified  users. 
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B.  2  Generalized  Effectiveness  Methodology  (GEM) 


GEM  provides  a  highly  flexible  computerized  capability  for  an?ly- 
scs  of  the  Availability  and  Dependability  terms  of  a  system  effectiveness 
model.  It  was  developed  to  permit  ease  and  rapidity  of  probabilistic 
analysis  of  complex  systems  characterized  by  on-line  repair  capability 
with  limited  resources,  long  mission  times,  and  extensive  redundancy. 
These  characteristics  generally  lead  to  a  more-complex  system  model 
when  one  i  ':cognizes  the  possibility  that  redundant  portions  of  a  system 
may  be  repaired  while  a  mission  is  in  process.  Ignoring  this  possibility 
generally  results  in  an  unduly  pessimistic  estimate  of  system  mission 
reliability.  However,  the  computations  required  by  this  more- complex 
system  model  are  prohibitive  for  systems  of  any  degree  of  complexity 
unless  machine  methods  are  employed.  GEM  was  developed  so  that  the 
system  models  incorporating  these  important  factors  can  be  compu¬ 
terized  and  exercised  with  minimal  effort  on  the  part  of  the  user.  GEM 
is  an  analytical  tool  which  provides  the  systems  Reliability/Maintain¬ 
ability/Availability/Effectiveness  analyst  with  the  means  to  analytically 
model  a  complex  system  against  equally  complex  multiphase  mission 
scenarios  and  to  perform  sensitivity  analyses  quickly  and  economically 
under  dynamically  changing  minion  revisions,  system  design  configura¬ 
tions,  changes  in  failure  definitions,  and  trade-off  studies. 


The  following  summary  of  GEM  capabilities  is  taken  from  the 
Handbook  of  Systems  Effectiveness  Models,  and  is  an  example  of  the 
type  of  information  provided: 


TITLE: 


Generalized  Effectiveness  Methodology 
(GEM) 


LANGUAGE: 


GEM  Language 


COMPUTER: 


CDC  6600 


DATE  OPERATIONAL:  August  1966 


DESCRIPTION: 


The  Generalized  Effectiveness  Method¬ 
ology  (GEM)  is  a  user-oriented  computer 
program  for  computing  one  or  more  of  a 
set  of  system  statistics  such  as  relia¬ 
bility,  availability,  mean  up  time,  mean 
down  time,  effective  failure  and  repair 
rates,  restore  time  distributions  and 
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B.  2  Generalized  Effectiveness  Methodology  (GEM) 


GEM  provides  a  highly  flexible  computerized  capability  for  analy¬ 
ses  of  the  Availability  and  Dependability  terms  of  a  system  effectiveness 
model.  It  was  developed  to  permit  ease  and  rapidity  of  probabilistic 
analysis  of  complex  systems  characterized  by  on-line  repair  capability 
with  limited  resources,  long  mission  times,  and  extensive  redundancy. 
These  characteristics  generally  lead  to  a  more-complex  system  model 
when  one  recognizes  the  possibility  that  redundant  portions  of  a  system 
may  be  repaired  while  a  mission  is  in  process.  Ignoring  this  possibility 
generally  results  in  an  unduly  pessimistic  estimate  of  system  mission 
reliability.  However,  the  computations  required  by  this  more- complex 
system  model  are  prohibitive  for  systems  of  any  degree  of  complexity 
unless  machine  methods  are  employed.  GEM  was  developed  so  that  the 
system  models  incorporating  these  important  factors  can  be  compu¬ 
terized  and  exercised  with  minimal  effort  on  the  part  of  the  user.  GEM 
is  an  analytical  tool  which  provides  the  systems  Reliability/Maintain¬ 
ability/Availability/Effectiveness  analyst  with  the  means  to  analytically 
model  a  complex  system  against  equally  complex  multiphase  mission 
scenarios  and  to  perform  sensitivity  an  *es  quickly  and  economically 
under  dynamically  changing  mission  re\  .  is,  system  design  configura¬ 
tions,  changes  in  failure  definitions,  ana  ..rade-off  studies. 

The  following  summary  of  GEM  capabilities  is  taken  from  the 
Handbook  of  Systems  Effectiveness  Models,  and  is  an  example  of  the 
type  of  information  provided: 

TITLE: 

LANGUAGE: 

COMPUTER: 

DATE  OPERATIONAL: 

DESCRIPTION: 


Generalized  Effectiveness  Methodology 
(GEM) 

GEM  Language 
CDC  6600 
August  1 966 

The  Generalized  Effectiveness  Method¬ 
ology  (GEM)  is  a  user -oriented  computer 
program  for  computing  one  or  more  of  a 
set  of  system  statistics  such  as  relia¬ 
bility,  availability,  mean  up  time,  mean 
down  time,  effective  failure  and  repair 
rates,  restore  time  distributions  and 
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COMMENTS: 


repairmen  atilizi.  *'n.  rhe  inputs 
required  are  th,» s  m  model  and  data 
in  the  form  of  parameters  of  distribu¬ 
tions  of  timer  to  failure  and  times  to 
repair  for  the  elementary  components 
of  the  system.  The  system  model  essen¬ 
tially  includes  the  reliability  configura¬ 
tion  of  the  system,  a  set  of  system  fail¬ 
ure  definitions,  and  logistic  information 
in  terms  of  repair  crews,  spares  pools 
and  priorities.  The  system  model  may 
include  any  reliability  structure.  For 
systems  without  repair,  the  elementary 
components  can  be  characterized  by  any 
one  cf  five  distributions  of  times  to  fail¬ 
ure:  exponential,  Weibull,  gamma,  log 
normal,  and  truncated  normal  distri¬ 
butions.  For  systems  with  repair,  dis¬ 
tributions  of  times  ic  failure,  repair,  or 
replacement  for  the  elementary  compo¬ 
nents  are  assumed  to  be  exponential 
only.  A  user- oriented  high  level  source 
language  is  used.  Inputs  written  in  the 
GEM  Language  are  used  by  the  GEM 
Compiler  to  generate  a  FORTRAN  pro¬ 
gram.  Depending  on  the  problem,  solu¬ 
tions  by  GEM  are  obtained  either  by 
combinatorial  methods  or  by  solving  a 
set  of  linear  differential  equations 
describing  the  system  state  probabilities, 
or  both.  Answers  are  given  in  the  forms 
of  tables  and  plots. 

The  original  GEM  program  was  updated 
aperiodically.  Documents  published  in 
1971  irclude  th^  Capability  Summary 
(TD  131),  User's  Manual  (TD  114), 
Reference  Manual  (TD  115),  and  Mathe¬ 
matics  Libary  Manual  (TD  116). 
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■SOUR  CE:  For  documents  or  further  information, 

contact  Naval  Electronics  Laboratory 
Center  (Code  4100),  San  Diego, 

California;  or  Naval  Ships  Engineering 
Center  (Code  6102C),  Hyattsville, 
Maryland. 

B.  3  Design  Disclosure  for  Systems  and  Equipment  (DDSE) 

Throughout  the  system  life  cycle  there  are  continuing  requirements 
for  various  types  of  effectiveness  analysis.  These  analytic  functions 
are  generally  performed  by  contractor  staff  organizations,  ij. -house 
Navy  groups,  or  independently  contracted  research  firms.  The  analyst 
expends  considerable  time  in  research  and  data  collection  before  he  can 
conduct  his  effectiveness  study.  Typically  he  is  forced  to  devote  more 
than  half  of  his  time  to  learning  the  system,  collecting  technical  data, 
and  defining  the  system  in  a  manner  required  for  analysis.  With  DDSE 
the  preparation  elements  of  an  analyst's  job  are  essentially  completed 
for  him,  and  the  corresponding  costs  can  be  averted.  Similarly,  the 
functions  of  system  engineering,  integrated  logistics  support  and  design 
engineering  are  greatly  benefited  by  the  increased  effectiveness  and 
efficiency  of  engineering/management  communication  provided  by  DDSE. 
Design  review  is  facilitated  through  consistency,  standardization,  com¬ 
pleteness  and  ease  of  interpretation  of  documentation,  and  evaluation  of 
the  findings  of  effectiveness  analysis  is  more-readily  carried  out  by 
managers  and  engineers  alike.  In  addition,  preparation  of  technical 
manuals*  and  optimal  location  of  test  points**  are  facilitated. 


See  MIL-M-24100A.  Manuals,  Orders  and  Other  Technical  Instruc¬ 
tions  for  Equipment  and  Systems  dated  15  June  1966,  which  calls  for 
symbolic  integrated  maintenance  manuals  (SIMM)  in  essentially  the 
same  format. 

u 

See  MIL-5TD- 1 326,  Test  Point  Selection  and  Interface  Requirements 
for  Equipment  Monitored  by  Shipboard  On-Line  Automatic  Test  Equip- 
ment  dated  16  January  1968,  which  requires  application  of  the  design 
disclosure  approach  to  test  point  selection. 
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MIL-HDBK-226  provides  the  basic  disclosure  for  DDSE.  A  useful 
introduction  is  the  "Manager's  Implementation  Guide  to  MIL-HDBK-226 
[June  1968].,  ,De3ign  Disclosure  for  /stems  Equipment. 

Four  basic  structures  comprise  the  DDSE  set.  They  form  the 
fundamental  vehicle  for  transmitting  design  information  between  Navy 
and  contractor  activities  jointly  working  together  for  one  objective,  the 
deployment  and  utilization  of  an  effective  system.  The  structures  are: 

•  Blocked  Schematics 

•  Detailed  Block  Diagrams 

•  Blocked  Texts 

•  Design  Outlines 

Varieties  of  these  format  types  are  used  to  map  design  configurations 

in  an  understandable  manner  during  the  entire  acquisition  cycle,  from 
concept  through  completion  of  production  models.  They  consolidate  in 
a  unified  scheme  basic  data  required  in  a  variety  of  engineering  disci¬ 
plines,  thus  becoming  a  single  source  document  containing  all  the 
required  design,  planning  and  effectiveness  information. 

Examples  of  the  four  basic  '  DSE  elements  (from  NELC  TD154) 
are  shown  in  Figures  B-l  through  B-5.  Figure  B-l  is  an  example  of 
a  blocked  schematic  for  a  voice  amplifier.  "System,”  assembly 
(PC  board},  and  circuit  boundaries  are  shown  as  shaded  areas  of  vary¬ 
ing  intensity.  To  simplify  functional  analysis,  the  blocked  schematic 
is  translated  into  a  detailed  block  diagram  (Figure  B-2)  which  shows 
only  the  functions.  Accompanying  the  blocked  schematics  and  detailed 
block  diagrams  are  blocked  texts  (Figures  B-3  and  B-4)  which  provide 
a  written  explanation  of  the  circuit  functions  and  operation  of  the  func¬ 
tions  and  assemblies.  Figure  B-5  illustrates  the  design  outline  which 
accompanies  the  other  DDSE  elements.  The  design  outline  is  a  short¬ 
hand  form  for  describing  system  operation,  requirements,  ard  depen¬ 
dencies.  Its  primary  purpose  is  to  illustrate  the  functional  dependencies 


Naval  Electronics  Laboratory  Center,  Manager's  Implementation  Guide 
to  MIL-HDBK-226  [^Tune  1  968j[,  Design  Disclosure  for  Systems  and 
Equipment,  Technical  Document  154,  14  December  1971.  (Qualified 
users  may  obtain  copies  from  Code  4100.  NELC,  San  Diego,  California) 
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Figure  B-3.  Blocked  text  to  accompany  blocked  schematic. 


Figure  B-4.  Blocked  text  to  accompany  detailed  block  diag 


of  a  system,  black  box,  assembly,  subassembly  or  circuit.  It  is  a 
logic  map  of  the  system  operation,  interdependencies,  input  require¬ 
ments,  output  requirements,  and  operating  sequences.  Its  primary 
value  is  system  analysis  rather  than  system  design;  it  describes 
event  dependencies  and  requirements  as  opposed  to  how  the  event  is 
made  to  occur.  The  design  outline  provides  an  important  input  to  aid 
in  structuring  the  effectiveness  model. 

In  addition  to  the  basic  DDSE  set,  various  types  of  supplemental 
data  keyed  to  the  basic  set  are  also  provided.  One  supplemental  sheet 
described  in  MIL-HDBK-226  is  the  system-equipment  description, 
which  presents  command-level  information  to  both  the  procuring  and 
producing  activities,  and  serves  as  a  working  paper  for  the  contracting 
office.  Since  the  system- equipment  description  is  of  standard  format, 
direct  overall  comparison  of  competing  designs  is  aided. 

Other  supplemental  data  sheets  which  may  be  required  include: 

•  System  effectiveness /system  value  studies 

•  Reliability/maintainability  analyses 

•  Tradeoff  studies 

•  Quality  assurance  reports  (MIL-Q-9858) 

•  Manufacturing  drawings 

•  Interface  control  drawings,  such  as  cabling  and  piping 

•  Maintenance  procedures 

•  Alignment/checkout  procedures 

•  Computer  software  documentation 

•  Human  factors  data,  including  operational  sequence 
diagrams  (MIL- H-46885) 

•  Integrated  Logistics  Support  (ILS)  data 


•  Packaging  data,  such  as  equipment  layouts,  module 
location  diagrams 

•  Cost  data 

•  Data  sources,  etc.  ,  as  required  separately  by  DD- 1423 

DDSE  is  prepared  with  the  users  in  mind,  the  users  comprising 
many  engineering  and  management  disciplines,  each  with  varied  back¬ 
grounds,  talents,  and  project  responsibilities.  All  pertinent  system 
criteria  and  design  information  are  addressed  on  the  applicable  DDSE 
forms.  The  design  disclosure  information  contained  on  the  DDSE  forms 
will  be  the  basis  for  many  decisions  having  major  impact  on  the  success 
or  failure  of  the  system. 

MIL- HDBK-226  defines  design  disclosure  formats  capable  of  pro¬ 
viding  any  level  of  system  detail  while  allowing  the  user  a  great  degree 
of  flexibility  in  their  preparation.  The  complete  net  of  formats 
described  in  that  handbook  include  the  following: 

•  Master  Block  Diagram  and  Text 

•  Master  Design  Outline 

•  Intermediate  Block  Diagram  and  Text 

•  Intermediate  Design  Outline 

•  Detailed  Block  Diagram  and  Text 

•  Power  Distribution  Diagram  and  Text 

•  Detailed  Design  Outline 

•  Blocked  Schematic  and  Text 

•  Front  Matter 

•  Supplemental  Information. 
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